luciano/vmware-go-kcl-v2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
146 commits 18 branches 1 tag 598 KiB
Commit graph

84 commits

Author SHA1 Message Date
vmwjc
8df326e926
Merge 486651702f into f6e79f1a2d 2024-06-20 04:08:02 +00:00
Yakkov Leng
3676eb410a fix: support go 1.21 & upgrade dependencies 
Signed-off-by: Yakkov Leng <yleng@vmware.com>
 2024-03-04 17:56:27 -08:00
John Calixto
486651702f fix: Move getRecords stop condition checking to top of loop 
Added tests to validate coordination of the lease renewal with the
record getting and processing routines. This brought to light the value
of checking the loop's "stop conditions" early in the loop instead of at
the end.

Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-04-08 12:03:24 -07:00
John Calixto
ec7c0b6f61 chore: Add test for renewLease 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-04-07 17:43:33 -07:00
Shiva Pentakota
4482696d95 fix: pass in ctx with cancel for renewLease 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-06 17:53:46 -07:00
vmwjc
16c5c53a30
Merge pull request #13 from calebstewart/fix/issue-5-empty-kinesisendpoint 
Automatically resolve default KinesisEndpoint
 2023-04-04 14:40:26 -07:00
John Calixto
c1f6b270ab chore: Remove extraneous err check 
After checking the scan result above this line, checking err here no
longer has any effect.

Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-04-04 14:15:59 -07:00
vmwjc
6120c11333
Merge pull request #10 from mrmonaghan/fix-infinite-worker-loop 
fixing infinite worker loop
 2023-04-04 14:08:59 -07:00
Shiva Pentakota
86d70940e6 fix: return err log in case of ErrLeaseNotAcquired 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-04 11:17:22 -07:00
Shiva Pentakota
4aebaf1ae0 feat: make lease renewal async 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-03 14:58:04 -07:00
Shiva Pentakota
02d4b44ff6 fix: add shutdown and leaseExpired error cases for checkpoint function 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-03-28 11:46:55 -07:00
John Calixto
987fada9d3 fix: Check token bucket corner cases correctly. 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-03-23 11:23:35 -07:00
Shiva Pentakota
a7c063b99c chore: add info logs in sleep case for kinesis backoff errors 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-03-22 12:22:26 -07:00
Shiva Pentakota
df16ef451c fix: use nanosecond precision in lease comparisons 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-02-13 17:56:11 -08:00
spentakota
fb17ec8bc6
Merge pull request #23 from vmware/spentakota_filNilError 
fix: add check for GetRecords error within callGetRecordsAPI
 2023-02-01 08:10:38 -08:00
Shiva Pentakota
04c5062ace fix: add check for GetRecords error within callGetRecordsAPI 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-02-01 08:00:49 -08:00
Shiva Pentakota
f879712f9d chore: log RemoveLeaseOwner errors with debug instead of error 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-31 10:18:15 -08:00
Shiva Pentakota
7d6b1c33d0 fix: add maxBytes per second getRecord check 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 16:28:22 -08:00
Shiva Pentakota
b5515931d1 fix: add hard cap maxRetries for getRecord errors 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 14:09:02 -08:00
Shiva Pentakota
66006caf89 fix: add getRecords TPS rate limiting 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 11:56:29 -08:00
John Calixto
3be57e8a74 Refactor in prep for testing rate limiting improvements 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-01-23 17:32:27 -08:00
Shiva Pentakota
599aa06ecd fix: add DeleteMetricMillisBehindLatest for error case 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-20 13:23:02 -08:00
Shiva Pentakota
e1425047a7 feat: Sending renewed lease metric 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-12 11:31:16 -08:00
Shiva Pentakota
09cc5896e9 chore: Adding periods to copyright comment to satisfy gofmt 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 19:03:29 -05:00
Shiva Pentakota
08b7fd9447 fix: catch DynamoDB Scan error when trying to scan nonexistent table/index in syncLeases() 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 18:07:22 -05:00
Caleb Stewart
e2a45c53c3 Automatically resolve default KinesisEndpoint 
This commit fixes #5 by returning `aws.EndpointNotFoundError` from the
endpoint resolver when no `KinesisEndpoint` is defined, which will
resolve the default AWS endpoint. This is the same process used by the
DynamoDB checkpointer to resolve the default endpoint.

Signed-off-by: Caleb Stewart <caleb.stewart94@gmail.com>
 2022-10-13 13:37:51 -04:00
Mike Monaghan
aab08b9050 fixing infinite worker loop 
Signed-off-by: Mike Monaghan <mike_monaghan@live.ca>
 2022-09-13 15:31:16 -06:00
Fabiano Arruda
c19ae1d605
Feature/fabiano commits (#2) 
* use cammel case

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix credential usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply lint against MD file

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade versions

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove calls to deprecated functions.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix cammel case usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* since deaggregation package was upgraded to sdk v2 makes sense to use it.

https://github.com/awslabs/kinesis-aggregation/pull/143#issuecomment-953308464
Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix format

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2022-01-06 20:13:32 -06:00
Tao Jiang
c02b7a85d4 Update unit tests 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-22 22:16:06 -06:00
Tao Jiang
86cc5a1a64 Update the libray reference path to new repo 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-21 13:49:47 -06:00
Fabiano Graças
f9ced84cbd improve gofmt 2021-12-20 21:21:15 -06:00
Fabiano Graças
7538535bff remove debug code 2021-12-20 21:21:15 -06:00
Fabiano Graças
a44513ef08 add parameters names in order to serve as suggestions and ignore explicitly bellow to avoid lint msgs. 2021-12-20 21:21:15 -06:00
Fabiano Graças
97c6633ea0 migrate to aws-sdk-go-v2 2021-12-20 21:21:15 -06:00
Fabiano Graças
0c204685a9 improve comments 2021-12-20 21:21:15 -06:00
Fabiano Graças
6372087bc3 removed due the new error handling https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling 2021-12-20 21:21:15 -06:00
Fabiano Arruda
7af9290557 Upgrade golang 1.17 (#98) 
* upgrade to golang 1.17

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

# Conflicts:
#	go.mod
#	go.sum

* improve after shell lint

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve after upgrade docker image (used by the build system)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove not needed variable

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply fixes after security scan (hmake test)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* add missing package after merge with latest master branch code.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve docker layer

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade packages

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2021-12-20 21:21:15 -06:00
Luca Rinaldi
0094ef5a69 improve log event (#93) 
* improve log event

Signed-off-by: lucarin91 <lucarin@protonmail.com>

* use %+v in template string

Signed-off-by: lucarin91 <lucarin@protonmail.com>
 2021-12-20 21:21:15 -06:00
Connor McKelvey
7de4607b71 Add support for lease stealing (#78) 
Fixes #4

Signed-off-by: Connor McKelvey <connormckelvey@gmail.com>
Signed-off-by: Ali Hobbs <alisuehobbs@gmail.com>
Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>

Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
4a642bfa2f Use application name as default enhanced fan-out consumer name (#91) 
* Use ApplicationName as default for EnhancedFanOutConsumerName

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add tests

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
ddcc2d0f95 Support enhanced fan-out feature (#90) 
* Implement enhanced fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add test cases

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Small adjustments in fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
909d1774a3 Add context to ErrLeaseNotAcquired (#87) 
* clientlibrary/checkpoint: convert ErrLeaseAcquired to struct

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* clientlibrary/checkpoint: add context to ErrLeaseNotAcquired

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Use errors.As to check for ErrLeaseNotAcquired error

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
adb264717b Fix naming convention (#85) 
Minor fix on constant naming convention.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
1044485392 Support Kinesis aggregation format (#84) 
Add support for Kinesis aggregation format to consume record
published by KPL.

Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.

Add cache entry and exit time.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
6ff3cd1b15 Fix retry logic for dynamodb (#83) 
Adding min/max retry and throttle delay for the retryer.
Also, increase the max retries to 10 which is inline with
dynamodb default retry count.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
f1982602ff Fix data race during checkpointing (#82) 
Make sure shard is locked during checkpointing.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
499e9cf1be Update aws go sdk and tests (#81) 
Update aws go sdk to the latest. Also, update
integration tests by publishing data using both
PutRecord and PutRecords.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
wgerges-discovery
384482169c Refactor getShardIDs (#70) 
* Refactor

* Use `nextToken` paramter as string.

Use `nextToken` paramter as string instead of pointer to match the original code base.

* Log the last shard token when failing.

* Use aws.StringValue to get the string pointer value.

Co-authored-by: Wesam Gerges <wesam.gerges.discovery@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
5dd53bf731 Add nil check before shutdown (#68) 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Kevin Burns
8f0d7bc8d8 Reduce log noise from found shards in worker event loop (#66) 
Signed-off-by: Kev Burns <kevburnsjr@gmail.com>
 2021-12-20 21:21:15 -06:00