luciano/vmware-go-kcl-v2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
114 commits 18 branches 1 tag 598 KiB
Commit graph

114 commits

This branch
This branch
All branches
Author SHA1 Message Date
Shiva Pentakota
15ecd714f8 fix: add missing ClaimRequest logic 2023-01-27 17:23:18 -08:00
spentakota
42881449ce
Merge pull request #21 from vmware/spentakota_callGetRecordsAPI 
fix: Handle ProvisionedThroughputExceededException throttling
 2023-01-24 17:12:39 -08:00
Shiva Pentakota
7d6b1c33d0 fix: add maxBytes per second getRecord check 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 16:28:22 -08:00
Shiva Pentakota
b5515931d1 fix: add hard cap maxRetries for getRecord errors 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 14:09:02 -08:00
Shiva Pentakota
66006caf89 fix: add getRecords TPS rate limiting 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 11:56:29 -08:00
spentakota
981dc2df11
Merge pull request #20 from vmware/jc/testable-getRecords 
Refactor in prep for testing rate limiting improvements
 2023-01-23 18:52:10 -08:00
John Calixto
3be57e8a74 Refactor in prep for testing rate limiting improvements 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-01-23 17:32:27 -08:00
spentakota
5e7aca6ab2
Merge pull request #18 from vmware/spentakota_deleteMetricMillis 
fix: add DeleteMetricMillisBehindLatest for error case
 2023-01-20 13:29:25 -08:00
Shiva Pentakota
599aa06ecd fix: add DeleteMetricMillisBehindLatest for error case 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-20 13:23:02 -08:00
spentakota
c5bc6c4ded
Merge pull request #17 from vmware/spentakota/sendLeaseRenewedMetric 
feat: Sending renewed lease metric
 2023-01-12 11:56:21 -08:00
Shiva Pentakota
e1425047a7 feat: Sending renewed lease metric 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-12 11:31:16 -08:00
spentakota
4afa8fec3e
Merge pull request #16 from vmware/spentakota_catchNilErrorSyncLeases 
fix: catch DynamoDB Scan error when trying to scan nonexistent table or index
 2022-12-19 08:34:02 -08:00
Shiva Pentakota
09cc5896e9 chore: Adding periods to copyright comment to satisfy gofmt 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 19:03:29 -05:00
Shiva Pentakota
08b7fd9447 fix: catch DynamoDB Scan error when trying to scan nonexistent table/index in syncLeases() 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 18:07:22 -05:00
Tao Jiang
c862165130
Update README.md 2022-01-06 20:24:58 -06:00
Fabiano Arruda
c19ae1d605
Feature/fabiano commits (#2) 
* use cammel case

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix credential usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply lint against MD file

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade versions

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove calls to deprecated functions.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix cammel case usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* since deaggregation package was upgraded to sdk v2 makes sense to use it.

https://github.com/awslabs/kinesis-aggregation/pull/143#issuecomment-953308464
Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix format

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2022-01-06 20:13:32 -06:00
Tao Jiang
c02b7a85d4 Update unit tests 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-22 22:16:06 -06:00
Tao Jiang
86cc5a1a64 Update the libray reference path to new repo 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-21 13:49:47 -06:00
Tao Jiang
aabb88c35b Update github workflow 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:43:57 -06:00
Tao Jiang
52f6fa57e6
Merge pull request #1 from vmware/upgrade/aws-sdk-to-v2 
Upgrade/aws sdk to v2
 2021-12-20 21:26:17 -06:00
Fabiano Graças
dd8ec33982 add build badge 2021-12-20 21:21:15 -06:00
Fabiano Graças
eaf26900ef add security scan 2021-12-20 21:21:15 -06:00
Fabiano Graças
0906af7baf improve lint 2021-12-20 21:21:15 -06:00
Fabiano Graças
fdbaa68af7 fix format 2021-12-20 21:21:15 -06:00
Fabiano Graças
7b0c198c0f setup go 1.17 2021-12-20 21:21:15 -06:00
Fabiano Graças
b20dd33ae2 upgrade packages 2021-12-20 21:21:15 -06:00
Fabiano Graças
7abd86ac6e add more branches 2021-12-20 21:21:15 -06:00
Fabiano Graças
f9ced84cbd improve gofmt 2021-12-20 21:21:15 -06:00
Fabiano Graças
6f0fbfe4c7 improve ci (remove hypermake, add makefile and github pipeline) 2021-12-20 21:21:15 -06:00
Fabiano Graças
7538535bff remove debug code 2021-12-20 21:21:15 -06:00
Fabiano Graças
2bf65b4386 update after tidy 2021-12-20 21:21:15 -06:00
Fabiano Graças
a44513ef08 add parameters names in order to serve as suggestions and ignore explicitly bellow to avoid lint msgs. 2021-12-20 21:21:15 -06:00
Fabiano Graças
97c6633ea0 migrate to aws-sdk-go-v2 2021-12-20 21:21:15 -06:00
Fabiano Graças
b537305690 add deaggregator v2 code straightly to the repo since it's seems they will not merge the PR soon. 2021-12-20 21:21:15 -06:00
Fabiano Graças
0c204685a9 improve comments 2021-12-20 21:21:15 -06:00
Fabiano Graças
6372087bc3 removed due the new error handling https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling 2021-12-20 21:21:15 -06:00
Fabiano Arruda
7af9290557 Upgrade golang 1.17 (#98) 
* upgrade to golang 1.17

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

# Conflicts:
#	go.mod
#	go.sum

* improve after shell lint

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve after upgrade docker image (used by the build system)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove not needed variable

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply fixes after security scan (hmake test)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* add missing package after merge with latest master branch code.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve docker layer

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade packages

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2021-12-20 21:21:15 -06:00
Máté Lang
18546df781 Added RS Zerolog logging adapter (#96) 
* Added RS Zerolog logging adapter

Signed-off-by: Mate Lang <mate.lang@welthee.com>

* Fixed pull request review comments

Signed-off-by: Mate Lang <mate.lang@welthee.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
5de70c0542 Rebuild and publish toolchain (#95) 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Luca Rinaldi
0094ef5a69 improve log event (#93) 
* improve log event

Signed-off-by: lucarin91 <lucarin@protonmail.com>

* use %+v in template string

Signed-off-by: lucarin91 <lucarin@protonmail.com>
 2021-12-20 21:21:15 -06:00
Connor McKelvey
7de4607b71 Add support for lease stealing (#78) 
Fixes #4

Signed-off-by: Connor McKelvey <connormckelvey@gmail.com>
Signed-off-by: Ali Hobbs <alisuehobbs@gmail.com>
Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>

Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
4a642bfa2f Use application name as default enhanced fan-out consumer name (#91) 
* Use ApplicationName as default for EnhancedFanOutConsumerName

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add tests

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
ddcc2d0f95 Support enhanced fan-out feature (#90) 
* Implement enhanced fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add test cases

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Small adjustments in fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
909d1774a3 Add context to ErrLeaseNotAcquired (#87) 
* clientlibrary/checkpoint: convert ErrLeaseAcquired to struct

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* clientlibrary/checkpoint: add context to ErrLeaseNotAcquired

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Use errors.As to check for ErrLeaseNotAcquired error

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
adb264717b Fix naming convention (#85) 
Minor fix on constant naming convention.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
1044485392 Support Kinesis aggregation format (#84) 
Add support for Kinesis aggregation format to consume record
published by KPL.

Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.

Add cache entry and exit time.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
6ff3cd1b15 Fix retry logic for dynamodb (#83) 
Adding min/max retry and throttle delay for the retryer.
Also, increase the max retries to 10 which is inline with
dynamodb default retry count.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
f1982602ff Fix data race during checkpointing (#82) 
Make sure shard is locked during checkpointing.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
499e9cf1be Update aws go sdk and tests (#81) 
Update aws go sdk to the latest. Also, update
integration tests by publishing data using both
PutRecord and PutRecords.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Mike Pye
10d6b28edf Use FieldLogger interface to represent logrus logger (#75) 
The FieldLogger interface is satisfied by either *Logger or *Entry.
Accepting this interface in place of the concrete *Logger type allows
users to inject a logger with some fields already set. For example, the
application developer might want all logging from the library to have a
`subsystem=kcl` field.

Signed-off-by: Mike Pye <mail@mdpye.co.uk>
 2021-12-20 21:21:15 -06:00