Nathan Leiby
d1473f894a
decode: s/type/decoder_msg_type
...
type is a really generally name. sometimes, it is used in other
libraries or various output as a valid field name. let's not make it a
reserved keyword that gets overriden by type == Kayvee
2017-08-17 14:36:54 -07:00
Xavi
4b4af77ba2
Merge pull request #9 from Clever/INFRA-2405-new-consumer-interface
...
Updated syslogparser library and added unit tests
2017-08-16 14:54:27 -07:00
Xavi Ramirez
e2f55b6d98
Updated syslogparser library and added unit tests
2017-08-16 20:58:30 +00:00
Nathan Leiby
035cc7fd83
Merge pull request #8 from Clever/INFRA-2263-kvroutes-bugfix
...
decode: json decodes dimensions as array of interface
2017-08-15 12:11:49 -07:00
Nathan Leiby
1bebeb5aa5
decode: json decodes dimensions as array of interface
...
Previously, this would result in a log line failing to extracting any
dimensions because
```golang
dimensions.([]string)
```
would fail because
```golang
reflect.TypeOf(dimensions)
```
was `[]interface{}`.
2017-08-15 11:47:01 -07:00
Nathan Leiby
375e4d1147
Merge pull request #7 from Clever/splitter-dont-filter-by-env
...
Splitter -- dont filter by env
2017-08-10 14:34:16 -07:00
Nathan Leiby
6039f69d86
(lint) go vet fixes
2017-08-10 14:28:13 -07:00
Nathan Leiby
5b8a86212d
remove DeployEnv (no longer needed)
2017-08-10 14:26:27 -07:00
Nathan Leiby
0c02e93821
splitter: don't filter by env
...
Filtering by env was an artifact of the previous way we originally had
setup our CWLogs subscriptions, assuming that we could have multiple
subscriptions per stream. :( You can't.
Let's instead allow any CWLogs subscription's logs to be valid, which is
needed for:
https://github.com/Clever/kinesis-cwlogs-splitter/pull/1
2017-08-10 14:23:03 -07:00
Xavi
16814bd597
Merge pull request #6 from Clever/INFRA-2405-new-consumer-interface
...
Made batchedconsumer interface production ready
2017-08-10 13:56:04 -07:00
Xavi Ramirez
55aeecddd7
Renamed IsEmpty to IsNil on SequencePair
2017-08-10 20:16:41 +00:00
Xavi Ramirez
45fad863d0
Made config object that's specific to BatcherManager
2017-08-10 20:11:24 +00:00
Xavi Ramirez
3a04dad4bf
Send single value instead of full config object to CheckpointManager
2017-08-10 20:01:07 +00:00
Xavi Ramirez
6102f9c752
Added comment to why a channel is taking a channel.
2017-08-10 19:56:17 +00:00
Xavi Ramirez
68a16cfe6a
Added an external go routine to ensure stale batches get flushed
2017-08-10 19:47:08 +00:00
Xavi Ramirez
27cce44140
Made constructors of private structs private
2017-08-10 19:21:46 +00:00
Xavi Ramirez
1632c81fc7
Use errors.New instead of fmt.Errorf
2017-08-10 19:20:55 +00:00
Xavi Ramirez
dd9da85055
Fixed type-o
2017-08-10 19:15:53 +00:00
Xavi Ramirez
b7743c9ea7
Added a more coherent shutdown pathway
2017-08-09 01:22:11 +00:00
Xavi Ramirez
1be812a887
Added less hacky way of downloading jars
2017-08-07 19:22:10 +00:00
Xavi Ramirez
de04a27799
Return NonKayveeError if log contians invalid or empty json
2017-08-07 19:20:56 +00:00
Xavi Ramirez
eb230b94f7
Added stats to base kinesis client
2017-08-07 19:20:56 +00:00
Xavi Ramirez
5c373fa7d8
Fixed unit tests
2017-08-07 19:20:56 +00:00
Xavi Ramirez
4809cdb4e6
Another refactor to simply and likely speed up code. New code allows checkpoint and readline messages to be interleaved.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
1bff01ff4f
Removed unused params and properties
2017-08-07 19:20:56 +00:00
Xavi Ramirez
734b46274f
Better coordinated checkpoints and readlines. Checkpoint function now returns all errors.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
b19c9297d8
Adjusted some locks
2017-08-07 19:20:56 +00:00
Xavi Ramirez
c814742afa
Another refactor
2017-08-07 19:20:56 +00:00
Xavi Ramirez
c5f75d6554
Fixed another race-condition. Ensure line is completely processed before allowing a checkpoint.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6a30e0eb8a
Huge refactor. Batchers no longer run in their own go-routines, which allowed us to a lot of concurrency and simplify model.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
ba951ff0da
Refactor to fix dead locks and race conditions.
2017-08-07 19:20:56 +00:00
Xavi Ramirez
29f68f77eb
Removed redundant retry logic
2017-08-07 19:20:56 +00:00
Xavi Ramirez
c6fe4cef37
Renamed variable from seq to pair
2017-08-07 19:20:56 +00:00
Xavi Ramirez
6e9457cbcf
Better propogated shutdown signal
2017-08-07 19:20:56 +00:00
Xavi Ramirez
873544ae78
Run benchmarks when building
2017-08-07 19:20:56 +00:00
Xavi Ramirez
266d7d620d
Fixed bug in decoder package
2017-08-07 19:20:56 +00:00
Xavi Ramirez
7a7716e824
Increased default read rate limit
2017-08-07 19:20:56 +00:00
Xavi Ramirez
04042290f5
Moved SequencePair to kcl package
2017-08-07 19:20:56 +00:00
Xavi
4c6d8453ef
Merge pull request #5 from Clever/INFRA-2405-new-consumer-interface
...
Implemented new consumer interface as well as example consumer
2017-07-24 13:12:43 -07:00
Xavi Ramirez
212ec45d88
Added writter unit tests and fixed a slew of concurrency bugs
2017-07-21 01:35:54 +00:00
Xavi Ramirez
939fc5406f
Batcher now returns error if bad config is provided
2017-07-21 01:33:59 +00:00
Xavi Ramirez
0cb14948a3
KCL more aggressively logs errors
2017-07-21 01:27:17 +00:00
Xavi Ramirez
ae37b57e61
Added IsEmpty to SequencePair
2017-07-21 01:25:02 +00:00
Xavi Ramirez
b0f769bfa7
Breaking change to KCL. Created Checkpointer interface to make testing easier.
2017-07-21 01:25:02 +00:00
Xavi Ramirez
164b9923be
Reworded docs to remove references to firehose. Renamed config variables to be clearer.
2017-07-21 01:25:02 +00:00
Xavi Ramirez
e8f40e607a
Added unit test for SequencePair IsLessThan function
2017-07-19 19:00:28 +00:00
Xavi Ramirez
1b2afcafc0
Renamed methods to be more generic and to remove reference to logs
2017-07-19 00:21:31 +00:00
Xavi Ramirez
f116c752f5
Added unit test to ensure correct error is returned
2017-07-19 00:20:20 +00:00
Xavi Ramirez
906f6b94fc
Made ReadBurstLimit relative to read rate limit
2017-07-18 19:53:25 +00:00
Xavi Ramirez
3f42cb5e4a
Created IsLessThan method to SequencePair to make code more readable
2017-07-18 19:52:26 +00:00
