Current, KCL doesn't release shard when returning on error
which causes the worker cannot get any shard because it has
the maximum number of shard already. This change makes sure
releasing shard when return.
update the log message.
Test:
Integration test by forcing error on reading shard to
simulate Kinesis Internal error and make sure the KCL
will not stop processing.
Jira CNA-1995
Change-Id: Iac91579634a5023ab5ed73c6af89e4ff1a9af564
After a few days of shard splitting, the parent shard will be
deleted by Kinesis system. KCL should ignore the error caused
by deleted parent shared and move on.
Test:
Manuall split shard on kcl-test stream in photon-infra account
Currently, shard3 is the parent shard of shard 4 and 5. Shard 3
has a parent shard 0 which has been deleted already. Verified
the test can run and not stuck in waiting for parent shard.
Jira CNA-2089
Change-Id: I15ed0db70ff9836313c22ccabf934a2a69379248
gas is now gosec. Need to update security scan and fix
security issue as needed.
No functional change.
Jira CNA-2022
Change-Id: I36f2a204114f3f13e2ed05579c04a9c89f528f9a
All source should be prepared in a manner that reflects
comments that VMware would be comfortable sharing with
the public.
Documentation only. No functional change.
Update the license to MIT to be consistent with approved
OSSTP product tracking ticket:
https://osstp.vmware.com/oss/#/upstreamcontrib/project/1101391
Jira CNA-1117
Change-Id: I3fe31f10db954887481e3b21ccd20ec8e39c5996
There might be verious reason for shard iterator to
expire, such as: not enough data in shard or process
even takes more than 5 minutes which cause shard
iterator not refreshing enough.
This change removes log.Fatal which causes panic.
Panic inside go routine will bring down the whole
app. Therefore, just log error and exit the go routine
instead.
Jira ID: CNA-1072
Change-Id: I34a8d9af7258f3ea75465e2245bbc25c2fafee35
cascade-kinesis-client will be used as a submodule of other projects,
so it should not have "src/vmware.com/cascade-kinesis-client" in
its path. To build this project locally, please manually create
the parent folders.
Change-Id: I8844e6a0e32aae65b28496915d8507e9fb1058c6
2021-12-20 21:15:15 -06:00
Renamed from src/vmware.com/cascade-kinesis-client/clientlibrary/worker/shard-consumer.go (Browse further)
