luciano/vmware-go-kcl-v2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
146 commits 18 branches 1 tag 598 KiB
Commit graph

84 commits

Author SHA1 Message Date
ravikiran-s
3c07ee37ac fix 2023-05-03 14:15:06 +05:30
ravikiran-s
6dec232363 nil checks 2023-05-03 13:56:49 +05:30
ravikiran-s
1babcd292f more logs 2023-05-03 12:19:33 +05:30
ravikiran-s
a372c4e9d0 dynamodbstreams adapter client with kcl library 2023-05-02 23:16:33 +05:30
Shiva Pentakota
4482696d95 fix: pass in ctx with cancel for renewLease 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-06 17:53:46 -07:00
vmwjc
16c5c53a30
Merge pull request #13 from calebstewart/fix/issue-5-empty-kinesisendpoint 
Automatically resolve default KinesisEndpoint
 2023-04-04 14:40:26 -07:00
John Calixto
c1f6b270ab chore: Remove extraneous err check 
After checking the scan result above this line, checking err here no
longer has any effect.

Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-04-04 14:15:59 -07:00
vmwjc
6120c11333
Merge pull request #10 from mrmonaghan/fix-infinite-worker-loop 
fixing infinite worker loop
 2023-04-04 14:08:59 -07:00
Shiva Pentakota
86d70940e6 fix: return err log in case of ErrLeaseNotAcquired 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-04 11:17:22 -07:00
Shiva Pentakota
4aebaf1ae0 feat: make lease renewal async 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-04-03 14:58:04 -07:00
Shiva Pentakota
02d4b44ff6 fix: add shutdown and leaseExpired error cases for checkpoint function 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-03-28 11:46:55 -07:00
John Calixto
987fada9d3 fix: Check token bucket corner cases correctly. 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-03-23 11:23:35 -07:00
Shiva Pentakota
a7c063b99c chore: add info logs in sleep case for kinesis backoff errors 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-03-22 12:22:26 -07:00
Shiva Pentakota
df16ef451c fix: use nanosecond precision in lease comparisons 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-02-13 17:56:11 -08:00
spentakota
fb17ec8bc6
Merge pull request #23 from vmware/spentakota_filNilError 
fix: add check for GetRecords error within callGetRecordsAPI
 2023-02-01 08:10:38 -08:00
Shiva Pentakota
04c5062ace fix: add check for GetRecords error within callGetRecordsAPI 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-02-01 08:00:49 -08:00
Shiva Pentakota
f879712f9d chore: log RemoveLeaseOwner errors with debug instead of error 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-31 10:18:15 -08:00
Shiva Pentakota
7d6b1c33d0 fix: add maxBytes per second getRecord check 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 16:28:22 -08:00
Shiva Pentakota
b5515931d1 fix: add hard cap maxRetries for getRecord errors 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 14:09:02 -08:00
Shiva Pentakota
66006caf89 fix: add getRecords TPS rate limiting 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-24 11:56:29 -08:00
John Calixto
3be57e8a74 Refactor in prep for testing rate limiting improvements 
Signed-off-by: John Calixto <jcalixto@vmware.com>
 2023-01-23 17:32:27 -08:00
Shiva Pentakota
599aa06ecd fix: add DeleteMetricMillisBehindLatest for error case 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-20 13:23:02 -08:00
Shiva Pentakota
e1425047a7 feat: Sending renewed lease metric 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2023-01-12 11:31:16 -08:00
Shiva Pentakota
09cc5896e9 chore: Adding periods to copyright comment to satisfy gofmt 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 19:03:29 -05:00
Shiva Pentakota
08b7fd9447 fix: catch DynamoDB Scan error when trying to scan nonexistent table/index in syncLeases() 
Signed-off-by: Shiva Pentakota <spentakota@vmware.com>
 2022-12-16 18:07:22 -05:00
Caleb Stewart
e2a45c53c3 Automatically resolve default KinesisEndpoint 
This commit fixes #5 by returning `aws.EndpointNotFoundError` from the
endpoint resolver when no `KinesisEndpoint` is defined, which will
resolve the default AWS endpoint. This is the same process used by the
DynamoDB checkpointer to resolve the default endpoint.

Signed-off-by: Caleb Stewart <caleb.stewart94@gmail.com>
 2022-10-13 13:37:51 -04:00
Mike Monaghan
aab08b9050 fixing infinite worker loop 
Signed-off-by: Mike Monaghan <mike_monaghan@live.ca>
 2022-09-13 15:31:16 -06:00
Fabiano Arruda
c19ae1d605
Feature/fabiano commits (#2) 
* use cammel case

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix credential usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply lint against MD file

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade versions

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove calls to deprecated functions.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix cammel case usage

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* since deaggregation package was upgraded to sdk v2 makes sense to use it.

https://github.com/awslabs/kinesis-aggregation/pull/143#issuecomment-953308464
Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* fix format

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2022-01-06 20:13:32 -06:00
Tao Jiang
c02b7a85d4 Update unit tests 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-22 22:16:06 -06:00
Tao Jiang
86cc5a1a64 Update the libray reference path to new repo 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-21 13:49:47 -06:00
Fabiano Graças
f9ced84cbd improve gofmt 2021-12-20 21:21:15 -06:00
Fabiano Graças
7538535bff remove debug code 2021-12-20 21:21:15 -06:00
Fabiano Graças
a44513ef08 add parameters names in order to serve as suggestions and ignore explicitly bellow to avoid lint msgs. 2021-12-20 21:21:15 -06:00
Fabiano Graças
97c6633ea0 migrate to aws-sdk-go-v2 2021-12-20 21:21:15 -06:00
Fabiano Graças
0c204685a9 improve comments 2021-12-20 21:21:15 -06:00
Fabiano Graças
6372087bc3 removed due the new error handling https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling 2021-12-20 21:21:15 -06:00
Fabiano Arruda
7af9290557 Upgrade golang 1.17 (#98) 
* upgrade to golang 1.17

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

# Conflicts:
#	go.mod
#	go.sum

* improve after shell lint

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve after upgrade docker image (used by the build system)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* remove not needed variable

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* apply fixes after security scan (hmake test)

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* add missing package after merge with latest master branch code.

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* improve docker layer

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

* upgrade packages

Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com>

Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
 2021-12-20 21:21:15 -06:00
Luca Rinaldi
0094ef5a69 improve log event (#93) 
* improve log event

Signed-off-by: lucarin91 <lucarin@protonmail.com>

* use %+v in template string

Signed-off-by: lucarin91 <lucarin@protonmail.com>
 2021-12-20 21:21:15 -06:00
Connor McKelvey
7de4607b71 Add support for lease stealing (#78) 
Fixes #4

Signed-off-by: Connor McKelvey <connormckelvey@gmail.com>
Signed-off-by: Ali Hobbs <alisuehobbs@gmail.com>
Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>

Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
4a642bfa2f Use application name as default enhanced fan-out consumer name (#91) 
* Use ApplicationName as default for EnhancedFanOutConsumerName

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add tests

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Ilia Cimpoes
ddcc2d0f95 Support enhanced fan-out feature (#90) 
* Implement enhanced fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Add test cases

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>

* Small adjustments in fan-out consumer

Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
 2021-12-20 21:21:15 -06:00
Aurélien Rainone
909d1774a3 Add context to ErrLeaseNotAcquired (#87) 
* clientlibrary/checkpoint: convert ErrLeaseAcquired to struct

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* clientlibrary/checkpoint: add context to ErrLeaseNotAcquired

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>

* Use errors.As to check for ErrLeaseNotAcquired error

Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
adb264717b Fix naming convention (#85) 
Minor fix on constant naming convention.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
1044485392 Support Kinesis aggregation format (#84) 
Add support for Kinesis aggregation format to consume record
published by KPL.

Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.

Add cache entry and exit time.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
6ff3cd1b15 Fix retry logic for dynamodb (#83) 
Adding min/max retry and throttle delay for the retryer.
Also, increase the max retries to 10 which is inline with
dynamodb default retry count.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
f1982602ff Fix data race during checkpointing (#82) 
Make sure shard is locked during checkpointing.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
499e9cf1be Update aws go sdk and tests (#81) 
Update aws go sdk to the latest. Also, update
integration tests by publishing data using both
PutRecord and PutRecords.

Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
wgerges-discovery
384482169c Refactor getShardIDs (#70) 
* Refactor

* Use `nextToken` paramter as string.

Use `nextToken` paramter as string instead of pointer to match the original code base.

* Log the last shard token when failing.

* Use aws.StringValue to get the string pointer value.

Co-authored-by: Wesam Gerges <wesam.gerges.discovery@gmail.com>
 2021-12-20 21:21:15 -06:00
Tao Jiang
5dd53bf731 Add nil check before shutdown (#68) 
Signed-off-by: Tao Jiang <taoj@vmware.com>
 2021-12-20 21:21:15 -06:00
Kevin Burns
8f0d7bc8d8 Reduce log noise from found shards in worker event loop (#66) 
Signed-off-by: Kev Burns <kevburnsjr@gmail.com>
 2021-12-20 21:21:15 -06:00