* Implement enhanced fan-out consumer
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
* Add test cases
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
* Small adjustments in fan-out consumer
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
Add support for Kinesis aggregation format to consume record
published by KPL.
Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.
Add cache entry and exit time.
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Add credential configuration for resources
Add credentials for Kinesis, DynamoDB and Cloudwatch. See the worker_test.go
to see how to use it.
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Add support for providing custom checkpointer
Provide a new constructor for adding checkpointer instead of alway using
default dynamodb checkpointer.
The next step is to abstract out the Kinesis into a generic stream API and
this will be bigger change and will be addressed in different PR.
Test:
Use the new construtor to inject dynamodb checkpointer and run the existing
tests.
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Add support for providing custom checkpointer
Provide a new constructor for adding checkpointer instead of alway using
default dynamodb checkpointer.
The next step is to abstract out the Kinesis into a generic stream API and
this will be bigger change and will be addressed in different PR.
Fix checkfmt error.
Test:
Use the new construtor to inject dynamodb checkpointer and run the existing
tests.
Signed-off-by: Tao Jiang <taoj@vmware.com>
All source should be prepared in a manner that reflects
comments that VMware would be comfortable sharing with
the public.
Documentation only. No functional change.
Update the license to MIT to be consistent with approved
OSSTP product tracking ticket:
https://osstp.vmware.com/oss/#/upstreamcontrib/project/1101391
Jira CNA-1117
Change-Id: I3fe31f10db954887481e3b21ccd20ec8e39c5996
The processing Kinesis gets stuck after splitting shard. The
reason is that the app doesn't do mandatory checkpoint.
KCL document states:
// When the value of {@link ShutdownInput#getShutdownReason()} is
// {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason#TERMINATE} it is required that you
// checkpoint. Failure to do so will result in an IllegalArgumentException, and the KCL no longer making progress.
Also, fix shard lease to prevent one host takes more shard than
its configuration allowed.
Jira CNA-1701
Change-Id: Icbdacaf347c7a67b5793647ad05ff93cca629741
cascade-kinesis-client will be used as a submodule of other projects,
so it should not have "src/vmware.com/cascade-kinesis-client" in
its path. To build this project locally, please manually create
the parent folders.
Change-Id: I8844e6a0e32aae65b28496915d8507e9fb1058c6
2021-12-20 21:15:15 -06:00
Renamed from src/vmware.com/cascade-kinesis-client/clientlibrary/worker/record-processor-checkpointer.go (Browse further)
