luciano/amazon-kinesis-client-go
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
204 commits 11 branches 1 tag 8.2 MiB
Commit graph

204 commits

This branch
This branch
All branches
Author SHA1 Message Date
Rafael
bfe2ac0f6d
Merge pull request #23 from Clever/INFRA-2764 
handle logs coming in from lambda functions
 2018-01-25 15:15:10 -08:00
Rafael Garcia
c255bf87f3 add test for default case 2018-01-25 15:13:11 +00:00
Rafael Garcia
10750e802b handle logs coming in from lambda functions 2018-01-25 01:23:03 +00:00
Nate
7d863db172
Merge pull request #22 from Clever/SYNC-114-go-1.9-upgrade 2017-11-09 08:48:35 -08:00
Nate Brennand
2a4e6fcc71 upgrade to 1.9 and convert to dep 2017-11-08 09:46:29 -08:00
Rafael
ebf2e2dc98
Merge pull request #21 from Clever/only-send-failed-log-lines-to-file 
write errors to stderr, failed log lines to file
 2017-11-03 11:05:31 -07:00
Rafael Garcia
7b18d1bab4 write failed log message to file 2017-11-03 18:03:39 +00:00
Rafael Garcia
945ed317c2 remove logger as parameter 2017-11-03 17:48:50 +00:00
Rafael Garcia
221743b3e2 write errors to stderr 2017-11-02 21:50:46 +00:00
Xavi
94aacdf833 Merge pull request #19 from Clever/rm-params-from-decoder 
Removed stringifyNested, renameESReservedFields, and minimumTimestamp params from decoder
 2017-09-26 14:53:56 -07:00
Xavi Ramirez
2ecbc081cd Removed stringifyNested, renameESReservedFields, and minimumTimestamp params from decoder. minimumTimestamp isn't needed anymore. stringifyNested and renameESReservedFields are very specific to the elasticsearch consumer and will be moved to the elasticsearch consumer. 2017-09-21 19:51:48 +00:00
Xavi
8d871591de Merge pull request #18 from Clever/fix-shutdown 
Fix shutdown
 2017-09-21 10:58:25 -07:00
Nathan Leiby
81fbb957c4 Merge pull request #17 from Clever/INFRA-2521-update-log-stream-parsing 
splitter: update log stream parsing
 2017-09-20 14:30:23 -07:00
Xavi Ramirez
b07ad4c1c2 Cleaned up shutdown code 2017-09-20 21:28:43 +00:00
Nathan Leiby
efa90d4481 splitter: update log stream parsing 
AWS changed the format of the log stream for AWS Batch logs. This
reflects that change.

---

http://docs.aws.amazon.com/batch/latest/userguide/job_states.html

Logs for RUNNING jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is jobDefinitionName/default/ecs_task_id (this format may change in the future).
After a job reaches the RUNNING status, you can programmatically retrieve its log stream name with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.
 2017-09-20 14:18:35 -07:00
Xavi Ramirez
7f2afe9d8f Send nil showdown signal immediately 2017-09-20 04:58:40 +00:00
mohit
32d4d33f00 Merge pull request #16 from Clever/always-log-counter-stats 
DefaultCounters to log 0 for counter stats
 2017-09-13 12:24:01 -07:00
mohit
22495acfe4 derpfix: s/black/blank 2017-09-13 19:22:58 +00:00
mohit
5b91cd80da DefaultCounters to log 0 for counter stats 
this makes sure that we can get metrics when things are not flowing
through one of the consumers
 2017-09-12 04:33:05 +00:00
mohit
b3d971e3ba Merge pull request #15 from Clever/typo-fix 
fix typo in error log
 2017-09-11 13:41:02 -07:00
mohit
8b44537934 fix typo in error log 2017-09-11 20:22:34 +00:00
Xavi
a559c33272 Merge pull request #14 from Clever/increase-default-read-rate 
Increased default read rate.
 2017-08-22 11:43:05 -07:00
Xavi Ramirez
f581bb4f6e Increased default read rate. 2017-08-22 18:37:47 +00:00
Xavi
7f39898b70 Merge pull request #13 from Clever/fix-checkpoint-with-archive-consumer 
Fix checkpoint with archive consumer
 2017-08-20 16:47:49 -07:00
Xavi Ramirez
09ea3282af Standardized error reporting 2017-08-20 23:42:27 +00:00
Xavi Ramirez
8f7cbf9952 Fixed case checkpoints would only occur on shutdown. The archive consumer fell victim of this bug. 2017-08-20 23:41:28 +00:00
Xavi
59a1fb7c1d Merge pull request #12 from Clever/fix-readline-edge-case 
Fixed edge case where the beginnings of messages would sometimes get …
 2017-08-20 11:35:12 -07:00
Xavi Ramirez
22915133fe Fixed edge case where the beginnings of messages would sometimes get chopped off as a result of creating a new buffered reader each time we'd read a line. 2017-08-20 03:28:50 +00:00
Xavi
d63effaddd Merge pull request #11 from Clever/properly-handle-shutdowns 
More properly handle errors from KCL
 2017-08-18 11:50:28 -07:00
Xavi Ramirez
b385278e04 More properly handle errors from KCL 2017-08-18 01:47:28 +00:00
Nathan Leiby
0a3335173a Merge pull request #10 from Clever/INFRA-2263-rename-type 
decode: s/type/decoder_msg_type
 2017-08-17 16:25:00 -07:00
Nathan Leiby
d1473f894a decode: s/type/decoder_msg_type 
type is a really generally name. sometimes, it is used in other
libraries or various output as a valid field name. let's not make it a
reserved keyword that gets overriden by type == Kayvee
 2017-08-17 14:36:54 -07:00
Xavi
4b4af77ba2 Merge pull request #9 from Clever/INFRA-2405-new-consumer-interface 
Updated syslogparser library and added unit tests
 2017-08-16 14:54:27 -07:00
Xavi Ramirez
e2f55b6d98 Updated syslogparser library and added unit tests 2017-08-16 20:58:30 +00:00
Nathan Leiby
035cc7fd83 Merge pull request #8 from Clever/INFRA-2263-kvroutes-bugfix 
decode: json decodes dimensions as array of interface
 2017-08-15 12:11:49 -07:00
Nathan Leiby
1bebeb5aa5 decode: json decodes dimensions as array of interface 
Previously, this would result in a log line failing to extracting any
dimensions because

```golang
dimensions.([]string)
```

would fail because

```golang
reflect.TypeOf(dimensions)
```

was `[]interface{}`.
 2017-08-15 11:47:01 -07:00
Nathan Leiby
375e4d1147 Merge pull request #7 from Clever/splitter-dont-filter-by-env 
Splitter -- dont filter by env
 2017-08-10 14:34:16 -07:00
Nathan Leiby
6039f69d86 (lint) go vet fixes 2017-08-10 14:28:13 -07:00
Nathan Leiby
5b8a86212d remove DeployEnv (no longer needed) 2017-08-10 14:26:27 -07:00
Nathan Leiby
0c02e93821 splitter: don't filter by env 
Filtering by env was an artifact of the previous way we originally had
setup our CWLogs subscriptions, assuming that we could have multiple
subscriptions per stream. :( You can't.

Let's instead allow any CWLogs subscription's logs to be valid, which is
needed for:
https://github.com/Clever/kinesis-cwlogs-splitter/pull/1
 2017-08-10 14:23:03 -07:00
Xavi
16814bd597 Merge pull request #6 from Clever/INFRA-2405-new-consumer-interface 
Made batchedconsumer interface production ready
 2017-08-10 13:56:04 -07:00
Xavi Ramirez
55aeecddd7 Renamed IsEmpty to IsNil on SequencePair 2017-08-10 20:16:41 +00:00
Xavi Ramirez
45fad863d0 Made config object that's specific to BatcherManager 2017-08-10 20:11:24 +00:00
Xavi Ramirez
3a04dad4bf Send single value instead of full config object to CheckpointManager 2017-08-10 20:01:07 +00:00
Xavi Ramirez
6102f9c752 Added comment to why a channel is taking a channel. 2017-08-10 19:56:17 +00:00
Xavi Ramirez
68a16cfe6a Added an external go routine to ensure stale batches get flushed 2017-08-10 19:47:08 +00:00
Xavi Ramirez
27cce44140 Made constructors of private structs private 2017-08-10 19:21:46 +00:00
Xavi Ramirez
1632c81fc7 Use errors.New instead of fmt.Errorf 2017-08-10 19:20:55 +00:00
Xavi Ramirez
dd9da85055 Fixed type-o 2017-08-10 19:15:53 +00:00
Xavi Ramirez
b7743c9ea7 Added a more coherent shutdown pathway 2017-08-09 01:22:11 +00:00