luciano/vmware-go-kcl-v2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
108 commits 18 branches 1 tag 598 KiB
Commit graph

108 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matteo Agius
c89ce3608b
ResetShardIterator added to rp interface (#2) 2023-02-07 20:25:45 -05:00
Matteo Agius
72d3b6f328
Merge pull request #1 from magiusdarrigo/fix-shard-shutdown-lease-release 
[MTCES-1062] call checkpointer nil in order to let workers know shard is expired
 2022-11-25 17:11:03 -06:00
magiusdarrigo
75831cfec5 call checkpointer nil in order to let workers know shard is expired 2022-11-25 17:07:01 -06:00
magiusdarrigo
09815b6898 changed back to magiudarrigo mod path 2022-10-27 13:04:44 -05:00
magiusdarrigo
66f1821f1b Revert "Revert "changed mod file path to my clone repo"" 
This reverts commit aa0a11286d.
 2022-10-27 12:58:37 -05:00
magiusdarrigo
d51f7a4d17 trying the go mod replace command with a v0.0.1 for my forked repo 2022-10-27 12:56:14 -05:00
magiusdarrigo
aa0a11286d Revert "changed mod file path to my clone repo" 
This reverts commit b602dfa20f.
 2022-10-27 12:49:02 -05:00
magiusdarrigo
b602dfa20f changed mod file path to my clone repo 2022-10-27 12:36:54 -05:00
Tao Jiang
c862165130
Update README.md 2022-01-06 20:24:58 -06:00
Fabiano Arruda
c19ae1d605
Feature/fabiano commits (#2) 
* use cammel case

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix credential usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply lint against MD file

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade versions

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove calls to deprecated functions.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix cammel case usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* since deaggregation package was upgraded to sdk v2 makes sense to use it.

https://github.com/awslabs/kinesis-aggregation/pull/143#issuecomment-953308464
Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix format

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2022-01-06 20:13:32 -06:00
Tao Jiang
c02b7a85d4 Update unit tests 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-22 22:16:06 -06:00
Tao Jiang
86cc5a1a64 Update the libray reference path to new repo 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-21 13:49:47 -06:00
Tao Jiang
aabb88c35b Update github workflow 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:43:57 -06:00
Tao Jiang
52f6fa57e6
Merge pull request #1 from vmware/upgrade/aws-sdk-to-v2 
Upgrade/aws sdk to v2
 2021-12-20 21:26:17 -06:00
Fabiano Graças
dd8ec33982 add build badge 2021-12-20 21:21:15 -06:00
Fabiano Graças
eaf26900ef add security scan 2021-12-20 21:21:15 -06:00
Fabiano Graças
0906af7baf improve lint 2021-12-20 21:21:15 -06:00
Fabiano Graças
fdbaa68af7 fix format 2021-12-20 21:21:15 -06:00
Fabiano Graças
7b0c198c0f setup go 1.17 2021-12-20 21:21:15 -06:00
Fabiano Graças
b20dd33ae2 upgrade packages 2021-12-20 21:21:15 -06:00
Fabiano Graças
7abd86ac6e add more branches 2021-12-20 21:21:15 -06:00
Fabiano Graças
f9ced84cbd improve gofmt 2021-12-20 21:21:15 -06:00
Fabiano Graças
6f0fbfe4c7 improve ci (remove hypermake, add makefile and github pipeline) 2021-12-20 21:21:15 -06:00
Fabiano Graças
7538535bff remove debug code 2021-12-20 21:21:15 -06:00
Fabiano Graças
2bf65b4386 update after tidy 2021-12-20 21:21:15 -06:00
Fabiano Graças
a44513ef08 add parameters names in order to serve as suggestions and ignore explicitly bellow to avoid lint msgs. 2021-12-20 21:21:15 -06:00
Fabiano Graças
97c6633ea0 migrate to aws-sdk-go-v2 2021-12-20 21:21:15 -06:00
Fabiano Graças
b537305690 add deaggregator v2 code straightly to the repo since it's seems they will not merge the PR soon. 2021-12-20 21:21:15 -06:00
Fabiano Graças
0c204685a9 improve comments 2021-12-20 21:21:15 -06:00
Fabiano Graças
6372087bc3 removed due the new error handling https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling 2021-12-20 21:21:15 -06:00
Fabiano Arruda
7af9290557 Upgrade golang 1.17 (#98) 
* upgrade to golang 1.17

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

# Conflicts:
#	go.mod
#	go.sum

* improve after shell lint

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve after upgrade docker image (used by the build system)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove not needed variable

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply fixes after security scan (hmake test)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* add missing package after merge with latest master branch code.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve docker layer

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade packages

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2021-12-20 21:21:15 -06:00
Máté Lang
18546df781 Added RS Zerolog logging adapter (#96) 
* Added RS Zerolog logging adapter

Signed-off-by: Mate Lang <mate.lang@welthee.com>

* Fixed pull request review comments

Signed-off-by: Mate Lang <mate.lang@welthee.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
5de70c0542 Rebuild and publish toolchain (#95) 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Luca Rinaldi
0094ef5a69 improve log event (#93) 
* improve log event

Signed-off-by: lucarin91 <lucarin@protonmail.com>

* use %+v in template string

Signed-off-by: lucarin91 <lucarin@protonmail.com>
 2021-12-20 21:21:15 -06:00
Connor McKelvey
7de4607b71 Add support for lease stealing (#78) 
Fixes #4

Signed-off-by: Connor McKelvey <connormckelvey@gmail.com>
Signed-off-by: Ali Hobbs <alisuehobbs@gmail.com>
Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>

Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
4a642bfa2f Use application name as default enhanced fan-out consumer name (#91) 
* Use ApplicationName as default for EnhancedFanOutConsumerName

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add tests

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
ddcc2d0f95 Support enhanced fan-out feature (#90) 
* Implement enhanced fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add test cases

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Small adjustments in fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
909d1774a3 Add context to ErrLeaseNotAcquired (#87) 
* clientlibrary/checkpoint: convert ErrLeaseAcquired to struct

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* clientlibrary/checkpoint: add context to ErrLeaseNotAcquired

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Use errors.As to check for ErrLeaseNotAcquired error

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
adb264717b Fix naming convention (#85) 
Minor fix on constant naming convention.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
1044485392 Support Kinesis aggregation format (#84) 
Add support for Kinesis aggregation format to consume record
published by KPL.

Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.

Add cache entry and exit time.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
6ff3cd1b15 Fix retry logic for dynamodb (#83) 
Adding min/max retry and throttle delay for the retryer.
Also, increase the max retries to 10 which is inline with
dynamodb default retry count.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
f1982602ff Fix data race during checkpointing (#82) 
Make sure shard is locked during checkpointing.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
499e9cf1be Update aws go sdk and tests (#81) 
Update aws go sdk to the latest. Also, update
integration tests by publishing data using both
PutRecord and PutRecords.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Mike Pye
10d6b28edf Use FieldLogger interface to represent logrus logger (#75) 
The FieldLogger interface is satisfied by either *Logger or *Entry.
Accepting this interface in place of the concrete *Logger type allows
users to inject a logger with some fields already set. For example, the
application developer might want all logging from the library to have a
`subsystem=kcl` field.

Signed-off-by: Mike Pye <mail@mdpye.co.uk>
 2021-12-20 21:21:15 -06:00
wgerges-discovery
384482169c Refactor getShardIDs (#70) 
* Refactor

* Use `nextToken` paramter as string.

Use `nextToken` paramter as string instead of pointer to match the original code base.

* Log the last shard token when failing.

* Use aws.StringValue to get the string pointer value.

Co-authored-by: Wesam Gerges <wesam.gerges.discovery@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
5dd53bf731 Add nil check before shutdown (#68) 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Kevin Burns
8f0d7bc8d8 Reduce log noise from found shards in worker event loop (#66) 
Signed-off-by: Kev Burns <kevburnsjr@gmail.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
f1935bc0ff Fix potentially delayed shutdown on shard sync (#64) 
ull-request #62 wrongly introduced an increased delay on
shutdown.

Before #62 the `stop` channel could be triggered while waiting for
`syncShard` milliseconds, so the function could return as soon as
`stop` was received.

However #62 changed this behavior by sleeping in the default case:
`stop` couldn't be handled right away anymore. Instead it was
handled after a whole new loop, potentially delaying shutdown by
minutes. (up to synchard * 1.5 ms).

This commit fixes that.

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
df60778d89 Re-org code for adding jittered delay for syncShard (#63) 
Minor update for the previous commit by removing duplicated code.
No functional change.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
43a936cab3 Issue 61/add shard sync jitter (#62) 
* Add a random number generator to Worker

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Add random jitter to the worker shard sync sleep

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Add random jitter in case syncShard fails

Fixes #61

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00