luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
616 commits 48 branches 124 tags 11 MiB
Commit graph

616 commits

This branch
This branch
All branches
Author SHA1 Message Date
dependabot[bot]
dd1e7aeeec Bump guava from 31.0.1-jre to 31.1-jre 
Bumps [guava](https://github.com/google/guava) from 31.0.1-jre to 31.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-09-06 13:46:30 -07:00
dependabot[bot]
c739f39a14 Bump logback-classic from 1.2.9 to 1.4.0 
Bumps [logback-classic](https://github.com/qos-ch/logback) from 1.2.9 to 1.4.0.
- [Release notes](https://github.com/qos-ch/logback/releases)
- [Commits](https://github.com/qos-ch/logback/compare/v_1.2.9...v_1.4.0)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-09-06 13:42:11 -07:00
dependabot[bot]
6c16b1c371 Bump awssdk.version from 2.17.108 to 2.17.267 
Bumps `awssdk.version` from 2.17.108 to 2.17.267.

Updates `kinesis` from 2.17.108 to 2.17.267

Updates `dynamodb` from 2.17.108 to 2.17.267

Updates `cloudwatch` from 2.17.108 to 2.17.267

Updates `netty-nio-client` from 2.17.108 to 2.17.267

Updates `sts` from 2.17.108 to 2.17.267

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:dynamodb
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudwatch
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:netty-nio-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-09-06 13:40:38 -07:00
Yu Zeng
2ae576a804 Preparation for v2.4.3 2022-09-06 13:25:44 -07:00
Yu Zeng
145c63a761 Upgrade dependencies 2022-09-06 13:17:18 -07:00
Jason Wang
32aaa66a50 Preparation for v2.4.2 
cr: https://code.amazon.com/reviews/CR-74125438
 2022-08-11 12:59:17 -07:00
Steven Shan
6c7d9ed662
Upgrade Lombok to version 1.18.24 (#972) 2022-08-11 11:06:54 -07:00
Adam Tapp
a344ee4d05 Added metric to record when a lease cannot be taken due to a ConditionalCheckFailedException 2022-03-31 09:15:41 -07:00
ava huang
862fabb266
add recommendation for customer (#937) 
Co-authored-by: Ava Huang <xinlyh@amazon.com>
 2022-03-30 18:36:31 -07:00
Yu Zeng
7f5eb9f34b Preparation for v2.4.1 - Fix pom version 2022-03-24 15:11:42 -07:00
Yu Zeng
b312c181e8 Preparation for v2.4.1 2022-03-24 14:35:18 -07:00
Brandon Dahler
1f15a43f03 Upgrade to rxjava3 2022-03-24 13:43:00 -07:00
Ravindranath Kakarla
5fa75ee3df
Preparation for v2.4.0 (#925) 
Co-authored-by: Ravindranath Kakarla <rnath@amazon.com>
 2022-03-02 16:10:46 -08:00
Kexin Hui
0a47137466 Upgrade GSR version to 1.1.9 2022-03-02 11:21:49 -08:00
dependabot[bot]
e722bbaa3e
Bump protobuf-java from 3.19.1 to 3.19.2 in /amazon-kinesis-client (#894) 
Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.19.1 to 3.19.2.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.19.1...v3.19.2)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-25 14:30:28 -08:00
Yu Zeng
fda91d9070 Preparing for release 2.3.10 2022-01-04 11:21:06 -08:00
Yu Zeng
a3e51d5595 Update the warning when lease taker failed to fetch the latest state of the lease 2022-01-04 10:34:27 -08:00
Yu Zeng
72aeac3819 Change log level from debug to warn when update stale leases 
Change log level from debug to warn when update stale leases
 2022-01-04 10:34:27 -08:00
Renju Radhakrishnan
a323272a97 Get latest counter before attempting a take to ensure take succeeds 2022-01-04 10:34:27 -08:00
Yu Zeng
b48de6d1bf
Change the scan interval of dependenbot to weekly (#888) 2021-12-30 11:50:33 -08:00
Yu Zeng
9c53c0ce59
Keep dependencies up-to-date (#879) 
* Test dependabot for upgrading dependencies automatically

* Add required field version in the configuration file

* Bump aws-java-sdk.version from 1.12.3 to 1.12.125

Bumps `aws-java-sdk.version` from 1.12.3 to 1.12.125.

Updates `aws-java-sdk-core` from 1.12.3 to 1.12.125
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.3...1.12.125)

Updates `aws-java-sdk-sts` from 1.12.3 to 1.12.125
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.3...1.12.125)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump gsr.version from 1.1.5 to 1.1.8

Bumps `gsr.version` from 1.1.5 to 1.1.8.

Updates `schema-registry-serde` from 1.1.5 to 1.1.8
- [Release notes](https://github.com/awslabs/aws-glue-schema-registry/releases)
- [Changelog](https://github.com/awslabs/aws-glue-schema-registry/blob/master/CHANGELOG.md)
- [Commits](https://github.com/awslabs/aws-glue-schema-registry/compare/v1.1.5...v1.1.8)

Updates `schema-registry-common` from 1.1.5 to 1.1.8
- [Release notes](https://github.com/awslabs/aws-glue-schema-registry/releases)
- [Changelog](https://github.com/awslabs/aws-glue-schema-registry/blob/master/CHANGELOG.md)
- [Commits](https://github.com/awslabs/aws-glue-schema-registry/compare/v1.1.5...v1.1.8)

---
updated-dependencies:
- dependency-name: software.amazon.glue:schema-registry-serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.glue:schema-registry-common
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-source-plugin from 3.0.1 to 3.2.1

Bumps [maven-source-plugin](https://github.com/apache/maven-source-plugin) from 3.0.1 to 3.2.1.
- [Release notes](https://github.com/apache/maven-source-plugin/releases)
- [Commits](https://github.com/apache/maven-source-plugin/compare/maven-source-plugin-3.0.1...maven-source-plugin-3.2.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-source-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump lombok from 1.16.20 to 1.18.22

Bumps [lombok](https://github.com/projectlombok/lombok) from 1.16.20 to 1.18.22.
- [Release notes](https://github.com/projectlombok/lombok/releases)
- [Changelog](https://github.com/projectlombok/lombok/blob/master/doc/changelog.markdown)
- [Commits](https://github.com/projectlombok/lombok/compare/v1.16.20...v1.18.22)

---
updated-dependencies:
- dependency-name: org.projectlombok:lombok
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-javadoc-plugin from 2.10.3 to 3.3.1

Bumps [maven-javadoc-plugin](https://github.com/apache/maven-javadoc-plugin) from 2.10.3 to 3.3.1.
- [Release notes](https://github.com/apache/maven-javadoc-plugin/releases)
- [Commits](https://github.com/apache/maven-javadoc-plugin/compare/maven-javadoc-plugin-2.10.3...maven-javadoc-plugin-3.3.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-javadoc-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump rxjava from 2.1.14 to 2.2.21

Bumps [rxjava](https://github.com/ReactiveX/RxJava) from 2.1.14 to 2.2.21.
- [Release notes](https://github.com/ReactiveX/RxJava/releases)
- [Changelog](https://github.com/ReactiveX/RxJava/blob/v2.2.21/CHANGES.md)
- [Commits](https://github.com/ReactiveX/RxJava/compare/v2.1.14...v2.2.21)

---
updated-dependencies:
- dependency-name: io.reactivex.rxjava2:rxjava
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-compiler-plugin from 3.8.0 to 3.8.1

Bumps [maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) from 3.8.0 to 3.8.1.
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](https://github.com/apache/maven-compiler-plugin/compare/maven-compiler-plugin-3.8.0...maven-compiler-plugin-3.8.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump guava from 29.0-jre to 31.0.1-jre

Bumps [guava](https://github.com/google/guava) from 29.0-jre to 31.0.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump protobuf-java from 3.11.4 to 3.19.1

Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.11.4 to 3.19.1.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.11.4...v3.19.1)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-gpg-plugin from 1.6 to 3.0.1

Bumps [maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) from 1.6 to 3.0.1.
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-1.6...maven-gpg-plugin-3.0.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump awssdk.version from 2.17.52 to 2.17.100

Bumps `awssdk.version` from 2.17.52 to 2.17.100.

Updates `kinesis` from 2.17.52 to 2.17.100

Updates `dynamodb` from 2.17.52 to 2.17.100

Updates `cloudwatch` from 2.17.52 to 2.17.100

Updates `netty-nio-client` from 2.17.52 to 2.17.100

Updates `sts` from 2.17.52 to 2.17.100

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:dynamodb
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudwatch
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:netty-nio-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump logback-classic from 1.2.7 to 1.2.9

Bumps logback-classic from 1.2.7 to 1.2.9.

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-surefire-plugin from 2.19.1 to 2.22.2

Bumps [maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.19.1 to 2.22.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.19.1...surefire-2.22.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump junit from 4.13.1 to 4.13.2

Bumps [junit](https://github.com/junit-team/junit4) from 4.13.1 to 4.13.2.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.13.1...r4.13.2)

---
updated-dependencies:
- dependency-name: junit:junit
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump commons-collections4 from 4.2 to 4.4

Bumps commons-collections4 from 4.2 to 4.4.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-collections4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump aws-java-sdk.version from 1.12.125 to 1.12.130

Bumps `aws-java-sdk.version` from 1.12.125 to 1.12.130.

Updates `aws-java-sdk-core` from 1.12.125 to 1.12.130
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.125...1.12.130)

Updates `aws-java-sdk-sts` from 1.12.125 to 1.12.130
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.125...1.12.130)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump awssdk.version from 2.17.100 to 2.17.101

Bumps `awssdk.version` from 2.17.100 to 2.17.101.

Updates `kinesis` from 2.17.100 to 2.17.101

Updates `dynamodb` from 2.17.100 to 2.17.101

Updates `cloudwatch` from 2.17.100 to 2.17.101

Updates `netty-nio-client` from 2.17.100 to 2.17.101

Updates `sts` from 2.17.100 to 2.17.101

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:dynamodb
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudwatch
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:netty-nio-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump commons-io from 2.7 to 2.11.0

Bumps commons-io from 2.7 to 2.11.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump slf4j-api from 1.7.25 to 1.7.32

Bumps [slf4j-api](https://github.com/qos-ch/slf4j) from 1.7.25 to 1.7.32.
- [Release notes](https://github.com/qos-ch/slf4j/releases)
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.25...v_1.7.32)

---
updated-dependencies:
- dependency-name: org.slf4j:slf4j-api
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump maven-failsafe-plugin from 2.19.1 to 2.22.2

Bumps [maven-failsafe-plugin](https://github.com/apache/maven-surefire) from 2.19.1 to 2.22.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.19.1...surefire-2.22.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-failsafe-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump jcommander from 1.72 to 1.81

Bumps [jcommander](https://github.com/cbeust/jcommander) from 1.72 to 1.81.
- [Release notes](https://github.com/cbeust/jcommander/releases)
- [Changelog](https://github.com/cbeust/jcommander/blob/master/CHANGELOG.md)
- [Commits](https://github.com/cbeust/jcommander/commits/1.81)

---
updated-dependencies:
- dependency-name: com.beust:jcommander
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump commons-lang3 from 3.8.1 to 3.12.0

Bumps commons-lang3 from 3.8.1 to 3.12.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-lang3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update doclint configuration to accommodate 3.x version of maven-javadoc-plugin

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yu Zeng <yuzen@amazon.com>
 2021-12-21 09:57:57 -08:00
ava huang
bedae95db9
Merge pull request #868 from QAQJ/consumer-level-metrics 
Adding a new metric: Application-level MillisBehindLatest
 2021-12-06 12:22:06 -08:00
Qilin Jin
e22079123c change the operation name for the metric 2021-12-02 10:52:21 -08:00
Qilin Jin
a2e0269826 add some comments to explain the metric scopes 2021-12-01 11:58:19 -08:00
Qilin Jin
66e5dfecb5 fix some naming convention issue 2021-12-01 11:38:21 -08:00
Sachin Sundar Pungampalayam Shanmugasundaram
25714f56c3
Preparing for release 2.3.9 (#867) 2021-11-22 16:21:55 -08:00
Qilin Jin
b43c14476b first commit for app-level mills_behind_latest metric 2021-11-22 16:10:31 -08:00
Sachin Sundar Pungampalayam Shanmugasundaram
d0fb6b5636
Update logback dependency. (#866) 
logback-classic version is updated to 1.2.7. The new version
includes security bug fixes.
 2021-11-22 13:46:19 -08:00
Sachin Sundar Pungampalayam Shanmugasundaram
9f6d27900c
Preparing for release 2.3.8 (#863) 2021-10-28 10:16:01 -07:00
Sachin Sundar Pungampalayam Shanmugasundaram
96e3345496
Revert commit for e638c17 and added new test (#861) 
* Revert commit for e638c17.

The change caused a regression for leases without owners.
Added unit tests.
 2021-10-27 16:45:04 -07:00
Ravindranath Kakarla
7503ec7105
Upgrade Glue schema registry from 1.1.4 to 1.1.5 (#860) 
Co-authored-by: Ravindranath Kakarla <rnath@amazon.com>
 2021-10-14 03:01:45 -04:00
Sachin Sundar Pungampalayam Shanmugasundaram
4f6691ae12
Preparing for release 2.3.7 (#859) 2021-10-11 14:41:42 -07:00
Sachin Sundar Pungampalayam Shanmugasundaram
6df221ebf2
Upgrade AWS SDK version to 2.17.52. (#858) 2021-10-08 18:05:56 -04:00
madagascar22
6fd59b326a
Fix to shutdown PrefetchRecordsPublisher in gracefull manner (#857) 
* Fix to shutdown PrefetchRecordsPublisher in gracefull manner

Previously when the lease expires PrefetchRecordsPublisher shutdown
the process forecefully by interupting the threads,
which lead to leak in apache http client connection
Now changed to code to shutdown the PrefetchRecordsPublisher
process in more gracefull manager and handled interrupted exception

* Fixed failing unit test

* Add awaitTerminationTimeoutMillis as paramter for PrefetchRecordsPublisher

Since clients can configure there own awaitTerminationTimeoutMillis,
add it as sepearate parameter with default value

* Fix setting interrupot status after shutdown

Co-authored-by: Monishkumar Gajendran <monishku@amazon.com>
 2021-10-06 17:35:09 -07:00
Sachin Sundar Pungampalayam Shanmugasundaram
e81969af59
Create DynamoDB tables on On-Demand billing mode by default. (#854) 
* Create DynamoDB tables on On-Demand billing mode by default.

This will enable KCL to create the DynamoDB tables - Lease and
ShardProgress with the On-Demand billing mode instead of provisioned
billing mode.

* Keep previous table creation function for backward compatibility.

* Added unit tests.

* Added more unit tests.
 2021-10-06 15:54:23 -07:00
Roberto Tyley
2447513de3
Fix DynamoDBLeaseTaker logging of available leases (#846) 
When `DynamoDBLeaseTaker` is deciding what leases to take to satisfy the
target lease-count of it's worker, it can either take *expired* leases
that haven't been updated by any other worker for a while, or failing
that, 'steal' a limited number (by default, 1) of leases from other
workers, in order to ensure its worker is gradually getting closer to
it's target.

It turns out that the log message it prints out when taking expired
leases is misleading. For instance, in this message - the leases
weren't stolen, they were taken from the available 'expired' leases:

```
Worker xxxxxx saw 68 total leases, 0 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases
```

...yet the message says there were 0 available leases, so how is that
possible? The truth is, there were 10 available expired leases, but the
log message was printing out the value of `expiredLeases.size()` - and
`expiredLeases` has been mutated over the course of the method, with
entries removed and added to `leasesToTake`:

0c5042dadf/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/dynamodb/DynamoDBLeaseTaker.java (L421)

The fix to logging in this commit just records `numAvailableLeases` at
the start of the method, so we can give an accurate log message:

```
Worker xxxxxx saw 68 total leases, 10 available leases, 6 workers. Target is 12 leases, I have 2 leases, I will take 10 leases
```
 2021-10-04 05:34:27 -07:00
Roberto Tyley
e638c1730a
Make use of Java 8 to simplify computeLeaseCounts() (#847) 
It's possible to remove quite a few lines of code from the
method using Java 8 features (a `groupingBy` Collector, etc).

Java 8 is the minimum supported version of the KCL since
https://github.com/awslabs/amazon-kinesis-client/pull/176 was
merged in July 2017.

See also:

https://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html#package.description
http://cr.openjdk.java.net/~briangoetz/lambda/lambda-translation.html
 2021-10-04 03:21:11 -07:00
Marcel Walden
4fe29ff444
Fixing typo is debug logs. (#842) 
maxLeasesToSteatAtOneTime -> maxLeasesToStealAtOneTime
 2021-10-04 03:18:09 -07:00
Ravindranath Kakarla
c5ce30f6f7
Emit Glue Schema Registry usage metrics (#855) 
* Emit Glue Schema Registry usage metrics

* Append KCL version to the app string.

Co-authored-by: Ravindranath Kakarla <rnath@amazon.com>
 2021-09-30 05:05:44 -07:00
rexcsn
e73a8a9f3a
Add configurable initial position for orphaned stream (#853) 
* Add default `orphanedStreamInitialPositionInStream` in `MultiStreamTracker`, so a custom initial position can be passed to Scheduler to initialize the orphaned stream's config
* Renamed `getDefaultStreamConfig` to `getOrphanedStreamConfig`
* Refactored `SchedulerTest` setup and implement `TestMultiStreamTracker` to test `MultiStreamTracker` interface default methods. Note that this is a workaround for using mockito 1.x to test default interface methods. mockito >= 2.7.13 supports Spy on interface directly, which can be used to test default methods without implementing a concrete class. However, mockito 2.x has a number of breaking changes, so future work will be needed to refactor unit tests and upgrade to mockito >= 2.7.13

Signed-off-by: Rex Chen <shuningc@amazon.com>
 2021-08-26 12:23:20 -07:00
ashwing
36ff9dc0b3
Merge pull request #843 from ashwing/scheduler_error_logging 
Better logging for scheduler exceptions
 2021-07-29 10:42:08 -07:00
Ashwin Giridharan
5857551edb Better logging for scheduler exceptions 2021-07-29 10:35:43 -07:00
Giancarlo Klemm Camilo
b98db8c1ce
Change hierarchicalShardSyncer getter to be public. (#841) 
Co-authored-by: Giancarlo Klemm Camilo <camilogi@amazon.com>
 2021-07-26 14:30:39 -07:00
Avinash Chowdary
1a51b0eace
Merge pull request #838 from AvinashChowdary/master 
Preparing for release 2.3.6
 2021-07-09 15:45:06 -07:00
Avinash Ravilla
35107a6188 Preparing for release 2.3.6 2021-07-09 13:25:39 -07:00
Avinash Chowdary
6adcbf49d5
Merge pull request #836 from AvinashChowdary/master 
Bumping aws sdk version
 2021-07-09 13:13:59 -07:00
Avinash Ravilla
9cc2074761 Bumping aws sdk version 2021-07-09 10:54:35 -07:00
Avinash Chowdary
3e2c6f3fdc
Merge pull request #835 from hhkkxxx133/schema_registry_upgrade 
Bump up Glue Schema Registry to 1.1.1
 2021-07-09 10:04:37 -07:00
Kexin Hui
75ff0ea584 Bump up Glue Schema Registry to 1.1.1 2021-07-08 18:32:24 -07:00
kmz
a25835678a
Update the Worker shutdown logic to make sure that the LeaseCleanup Manager also terminates all the threads that it has started (#817) 
Co-authored-by: kmz <kmz@users.noreply.github.com>
 2021-06-30 17:32:16 -07:00