Commit graph

12 commits

Author SHA1 Message Date
Tao Jiang
c05bfb7ac8 KCL: Fixing checkpoint operation
This change fixed the bug of not finding checkpoint when process
restart. It also adds missing call to record processor for notifying
the shard info and checkpoint when application first started.

Test:
Run hmake test and verify the log.

Change-Id: I4bdf21ac10c5ee988a0860c140991f7d05975541
2021-12-20 21:13:20 -06:00
Tao Jiang
a323d2fd51 KCL: Implement Worker
This is the core part of KCL by implementing worker.
It has exactly the same interface as Amazon's KCL. Internally,
it uses code from GoKini in order to get the library
functionaly quickly.

This is a working version. The test code worker_test.go
shows how to use this library.

Dynamic resharding feature is out of the scope of M4.

Test:

1. A Kinesis stream named "kcl-test" has been created under photon-infra
account.
2. Download your AWS Credential from IAM user page.
3. Modify the worker_test.go to fill in your aws credential.
4. hmake test

Jira CNA-637

Change-Id: I886d255bab9adaf7a13bca11bfda51bedaacaaed
2021-12-20 21:13:20 -06:00
Tao Jiang
1969713863 KCL: Fix unit test
Fix code bug for removing cyclic dependency and fix unit test.

Test:
hmake test

Change-Id: Ib4d4ba416d0133542e6459459ddf43079ff53ab8
2021-12-20 21:13:20 -06:00
Tao Jiang
425daf70ce KCL: Implement Shard Lease (part 1/2)
This is the first part of implementing shard lease for Kinesis
Client library. It creates dynamoDB table for managing
Kinesis stream shard lease.

https://jira.eng.vmware.com/browse/CNA-636

Adjust error code value range.

Change-Id: I16565fa15332843101235fb14545ee69c2599f2f
2021-12-20 21:13:20 -06:00
Tao Jiang
702335374c [WIP] KCL: create configuration and interface for kinesis client library
This is to create configuration and client interface in order to give
user an overview on how the Kinesis client library works.

In order not to reinvent wheel, the api is designed closely aligned with
Amazon Kinesis Client Library in Java.

add errors.
remove @throws and use @error instead.

https://jira.eng.vmware.com/browse/CNA-614

Change-Id: I78a269b328c14df37f878eccef192ff022a669cc
2021-12-20 21:13:20 -06:00
Tao Jiang
7b35571d9e CKL: Initial version for starting repository
Change-Id: Ic8dc81a437d9c72e841048359745e245c6c4895b
2021-12-20 21:13:11 -06:00
Tao Jiang
f7763e6d8b Initial empty repository 2021-12-20 21:11:14 -06:00
VMware GitHub Bot
157c2a8b3e Add CONTRIBUTING template 2021-12-20 21:11:09 -06:00
VMware GitHub Bot
2afd07ed52 Adding .gitignore template 2021-11-30 23:05:17 +08:00
VMware GitHub Bot
65c4e3fc0b Add Code of Conduct template 2021-11-30 23:05:17 +08:00
VMware GitHub Bot
5de8e31499 Add CONTRIBUTING template 2021-11-30 23:05:16 +08:00
VMware GitHub Bot
7ad2bfc721 Add README template 2021-11-30 23:05:15 +08:00