Commit graph

2 commits

Author SHA1 Message Date
Tao Jiang
869a8e4275 KCL: Add support for handling shard split
Add support for handling child/parent shard. When processing
child shard, it has to wait until parent shard finished before
processing itself.

Change-Id: I8bbf104c22ae93409d856be9c6829988c1b2d7eb
2021-12-20 21:13:20 -06:00
Tao Jiang
a323d2fd51 KCL: Implement Worker
This is the core part of KCL by implementing worker.
It has exactly the same interface as Amazon's KCL. Internally,
it uses code from GoKini in order to get the library
functionaly quickly.

This is a working version. The test code worker_test.go
shows how to use this library.

Dynamic resharding feature is out of the scope of M4.

Test:

1. A Kinesis stream named "kcl-test" has been created under photon-infra
account.
2. Download your AWS Credential from IAM user page.
3. Modify the worker_test.go to fill in your aws credential.
4. hmake test

Jira CNA-637

Change-Id: I886d255bab9adaf7a13bca11bfda51bedaacaaed
2021-12-20 21:13:20 -06:00