* Implement enhanced fan-out consumer
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
* Add test cases
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
* Small adjustments in fan-out consumer
Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
Add support for Kinesis aggregation format to consume record
published by KPL.
Note: current implementation need to checkpoint the whole batch
of the de-aggregated records instead of just portion of them.
Add cache entry and exit time.
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Remove MonitoringConfiguration and export no-op service
MonitoringConfiguration is not needed anymore as the user directly
implements its monitoring service or use one the default constructors.
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Provide a constructor for CloudWatchMonitoringService
Unexport all fields
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Provide a constructor to PrometheusMonitoringService
Unexport fields
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Remove all CloudWatch specific-stuff from config package
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* NewWorker accepts a metrics.MonitoringService
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Fix tests
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Add WithMonitoringService to config
Instead of having an additional parameter to NewWorker so that the
user can provide its own MonitoringService, WithMonitoringService
is added to the configuration. This is much cleaner and remains
in-line with the rest of the current API.
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Fix tests after introduction of WithMonitoringService
Also, fix tests that should have been fixed in earlier commits.
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Move Prometheus into its own package
Also rename it to prometheus.MonitoringService to not have to repeat
Prometheus twice when using.
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Move CloudWatch metrics into its own package
Also rename it to cloudwatch.MonitoringService to not have to repeat
Cloudwatch twice when using.
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Remove references to Cloudwatch in comments
Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Release shard lease after shutdown
Currently, only local cached shard info has been removed when worker losts the
lease. The info inside checkpointer (dynamoDB) is not removed. This causes
lease has been hold until the lease expiration and it might take too long
for shard is ready for other worker to grab. This change release the lease
in checkpointer immediately.
The user need to ensure appropriate checkpointing before return from
Shutdown callback.
Test:
updated unit test and integration test to ensure only the shard owner
has been wiped out and leave the checkpoint information intact.
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Add code coverage reporting
Add code coverage reporting for unit test.
Signed-off-by: Tao Jiang <taoj@vmware.com>
Update the unit test and move integration test under test folder.
Update retry logic by switching to AWS's default retry.
Signed-off-by: Tao Jiang <taoj@vmware.com>
