luciano/amazon-kinesis-client
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
616 commits 48 branches 124 tags 11 MiB
Commit graph

616 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nakul Joshi
409a8f13cd add spotless 2024-05-21 01:27:20 -07:00
Nakul Joshi
c5e4fa2de0 remove commented imports 2024-05-21 01:27:20 -07:00
dependabot[bot]
a8abbea29f Bump org.apache.maven.plugins:maven-surefire-plugin from 3.1.2 to 3.2.5 
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.1.2 to 3.2.5.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.1.2...surefire-3.2.5)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-20 16:21:00 -07:00
dependabot[bot]
2648123a79 Bump org.apache.maven.plugins:maven-checkstyle-plugin 
Bumps [org.apache.maven.plugins:maven-checkstyle-plugin](https://github.com/apache/maven-checkstyle-plugin) from 3.3.0 to 3.3.1.
- [Commits](https://github.com/apache/maven-checkstyle-plugin/compare/maven-checkstyle-plugin-3.3.0...maven-checkstyle-plugin-3.3.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-checkstyle-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-20 16:19:38 -07:00
vincentvilo-aws
023f6be251
Modify backwards compatibility script to ignore synthetic access methods (#1330) 
* modify backwards compatibility script to ignore synthetic access methods

* ignore synthetic access methods starting at access\$000
 2024-05-20 13:06:24 -07:00
nakulj
71f31e0a11
Remove unnecessary lambda (#1329) 
> Returning a lambda from a helper method or saving it in a constant is unnecessary; prefer to implement the functional interface method directly and use a method reference instead.

https://errorprone.info/bugpattern/UnnecessaryLambda
 2024-05-09 15:22:45 -07:00
Aravinda Kidambi Srinivasan
16e8404dc4
Fix a race condition between ShardConsumer shutdown and initialization (#1319) 
* Fix a race condition between ShardConsumer shutdown and initialization

When Kinesis shards have no data, there can be a race condition where
the shard-end record processing from RecordProcessorThread
interleaves with Scheduler performing initialization.
This leads to ShardConsumer making incorrect state transition
during initialization (moves from PROCESSING -> SHUTTING_DOWN) state
and during shutdown handling it moves from SHUTTING_DOWN -> SHUTDOWN_COMPLETE
without running the ShutdownTask.

This can cause the ShardConsumer to not perform proper shutdown
processing that is required for a child shard processing
to be unblocked. So the child shard could be blocked forever unless the
lease for the parent shard moves to a new worker and that worker does
not run into the race condition.

This patch fixes the race condition as follows:

The intializationComplete invocation is not needed after
needsInitialization has been set to false. Because initializationComplete
is mean to perform initialization in an async manner, but once
its done, the async task is a no-op in happy-path, but it can
perform incorrect state transition during a race condition.
 2024-05-02 14:54:59 -07:00
chenylee-aws
69cf5996c5
Honor lease sync on app bootstrap (#1325) 2024-05-02 14:43:03 -07:00
kcl-release-automation-bot
35fc72b2c8
Add snapshot to version 2.6.1-SNAPSHOT (#1324) 2024-05-01 17:54:42 -07:00
kcl-release-automation-bot
89a90e34e2
Preparation for v2.6.0 (#1323) 2024-05-01 17:48:25 -07:00
lucienlu-aws
ab24b66039
Skip cross account tests if no cross account credentials are provided (#1321) 2024-04-30 16:56:05 -07:00
Brendan Lynch
34fe58c492
Get unassigned leases in leasesToTake (#1320) 
Consider all null leases as a possible lease to take alongside expired leases

---------

Co-authored-by: Brendan Lynch <brenplyn@amazon.com>
 2024-04-30 15:56:31 -07:00
furq-aws
ec34ed1def
Internally construct and use stream ARNs for all streams in multi-stream mode (#1318) 2024-04-30 14:19:58 -07:00
lucienlu-aws
c12cee2a1b
Add additional integration tests for multistream and cross account (#1313) 2024-04-30 11:16:37 -07:00
lucienlu-aws
96be30b3e7
Change agedFailoverTimeMultiplier config to doPriorityLeaseTaking (#1317) 
* Change agedFailoverTimeMultiplier config to doPriorityLeaseTaking
 2024-04-22 10:35:08 -07:00
vincentvilo-aws
7d55b7a77f
Create script to check for backwards compatibility (#1316) 
* create script to check for backwards compatibility

* move scripts directory to .github folder

* add execute permissions to script

* moved the 'continue' call to the find_removed_methods() function

* add function to check if minor release is being performed

* Upgrade version to 2.6.0-SNAPSHOT
 2024-04-19 14:01:04 -07:00
furq-aws
e9990190cc
Update RetrievalFactory implementations to utilize the StreamIdentifier field of StreamConfig (#1291) 2024-04-15 11:14:13 -07:00
vincentvilo-aws
969341130a
move shutdownComplete call to ShardConsumer (#1308) 
* move shutdownComplete call to ShardConsumer
 2024-04-10 14:10:52 -07:00
lucienlu-aws
981899499f
Expose veryOldLeaseDurationMultiplier in LeaseManagementConfig (#1307) 
* Expose veryOldLeaseDurationMultiplier in LeaseManagementConfig as agedFailoverTimeMultiplier
 2024-04-10 11:24:53 -07:00
vincentvilo-aws
7f1f243676
Correcting the behavior of gracefulShutdown (#1302) 
* modify ShutdownTask to call shutdownComplete for graceful shutdown

* add test to verify ShutdownTask succeeds regardless of shutdownNotification

* change access level for finalShutdownLatch to NONE

* remove unused variable in GracefulShutdownCoordinator

* make comment more concise

* move waitForFinalShutdown method into GracefulShutdownCoordinator class

* cleanup call method of GracefulShutdownCoordinator

* modify waitForFinalShutdown to throw InterruptedException
 2024-04-03 12:42:26 -07:00
kcl-release-automation-bot
581d713815
Add snapshot to version 2.5.9-SNAPSHOT (#1303) 2024-04-02 08:09:22 -07:00
chenylee-aws
24774bc2e3
Clean up streams from currentStreamConfigMap (#1273) 2024-04-01 14:53:16 -07:00
kcl-release-automation-bot
bf5ab60f4b
Preparation for v2.5.8 (#1290) 2024-03-27 15:19:20 -07:00
dependabot[bot]
44beda5cb7
Upgrade awssdk.version from 2.25.3 to 2.25.11 (#1278) 
Bumps `awssdk.version` from 2.25.3 to 2.25.11.

Updates `software.amazon.awssdk:kinesis` from 2.25.3 to 2.25.11

Updates `software.amazon.awssdk:dynamodb` from 2.25.3 to 2.25.11

Updates `software.amazon.awssdk:cloudwatch` from 2.25.3 to 2.25.11

Updates `software.amazon.awssdk:netty-nio-client` from 2.25.3 to 2.25.11

Updates `software.amazon.awssdk:sts` from 2.25.3 to 2.25.11

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:dynamodb
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudwatch
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:netty-nio-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-27 10:09:21 -07:00
dependabot[bot]
38b8eb5fa2
Upgrade org.apache.maven.plugins:maven-gpg-plugin from 3.1.0 to 3.2.1 (#1279) 
Bumps [org.apache.maven.plugins:maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) from 3.1.0 to 3.2.1.
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.1.0...maven-gpg-plugin-3.2.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-27 10:08:41 -07:00
dependabot[bot]
c6b5d6872d
Upgrade org.apache.commons:commons-lang3 from 3.12.0 to 3.14.0 (#1280) 
Bumps org.apache.commons:commons-lang3 from 3.12.0 to 3.14.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-lang3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-27 10:07:56 -07:00
dependabot[bot]
fcfa3ec08f
Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.6.3 (#1282) 
Bumps [org.apache.maven.plugins:maven-javadoc-plugin](https://github.com/apache/maven-javadoc-plugin) from 3.5.0 to 3.6.3.
- [Release notes](https://github.com/apache/maven-javadoc-plugin/releases)
- [Commits](https://github.com/apache/maven-javadoc-plugin/compare/maven-javadoc-plugin-3.5.0...maven-javadoc-plugin-3.6.3)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-javadoc-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-27 10:07:03 -07:00
laxeo
e9b810ba07
Reuse 'ShardSyncTaskManager' instance for existing stream to avoid duplicate enqueue of 'ShardSyncTask' (#1277) 
Co-authored-by: Tongqing Zhang <tqzhang@amazon.com>
 2024-03-26 13:13:39 -07:00
kcl-release-automation-bot
f205673fee
Preparation for v2.5.7 (#1287) 2024-03-19 11:09:32 -07:00
Eric Meisel
a95aa9e39e
Update PollingConfig maxRecords method to return PollingConfig (#1275) 2024-03-15 14:11:22 -07:00
dependabot[bot]
ddff314020
Bump commons-io:commons-io from 2.11.0 to 2.15.1 (#1236) 
Bumps commons-io:commons-io from 2.11.0 to 2.15.1.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-15 10:38:07 -07:00
dependabot[bot]
f6df50ef6e
Bump org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1 (#1189) 
Bumps [org.apache.maven.plugins:maven-resources-plugin](https://github.com/apache/maven-resources-plugin) from 3.3.0 to 3.3.1.
- [Release notes](https://github.com/apache/maven-resources-plugin/releases)
- [Commits](https://github.com/apache/maven-resources-plugin/compare/maven-resources-plugin-3.3.0...maven-resources-plugin-3.3.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-14 15:51:25 -07:00
dependabot[bot]
f16217f1a9
Bump maven-surefire-plugin from 2.22.2 to 3.1.2 (#1139) 
Bumps [maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.1.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.1.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-14 15:49:36 -07:00
dependabot[bot]
63ba724df7
Bump maven-failsafe-plugin from 2.22.2 to 3.1.2 (#1138) 
Bumps [maven-failsafe-plugin](https://github.com/apache/maven-surefire) from 2.22.2 to 3.1.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-2.22.2...surefire-3.1.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-failsafe-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-14 15:39:55 -07:00
dependabot[bot]
35b7e73514
Bump maven-gpg-plugin from 3.0.1 to 3.1.0 (#1125) 
Bumps [maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) from 3.0.1 to 3.1.0.
- [Commits](https://github.com/apache/maven-gpg-plugin/compare/maven-gpg-plugin-3.0.1...maven-gpg-plugin-3.1.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-14 15:28:58 -07:00
kcl-release-automation-bot
1675f0a297
Preparation for v2.5.6 (#1272) 2024-03-11 08:58:55 -07:00
vincentvilo-aws
1280325c20
add functionality to retry an InvalidArgumentException (#1270) 
* add functionality to retry an InvalidArgumentException with a new iterator

* include DEFAULT_MAX_RECORDS value in IllegalArgumentException messages
 2024-03-08 13:37:25 -08:00
Brendan Lynch
63e0fe7537
Adding snapshot for 2.5.6-SNAPSHOT (#1271) 
---------

Co-authored-by: Brendan Lynch <brenplyn@amazon.com>
 2024-03-08 09:31:07 -08:00
Brendan Lynch
b4c2c6c947
Upgrade ch.qos.logback:logback-classic dependency from 1.3.12 to 1.3.14, awssdk.version from 2.20.43 to 2.25.3 aws-java-sdk.version from 1.12.405 to 1.12.668 gsr.version from 1.1.17 to 1.1.19 
* Upgrade ch.qos.logback:logback-classic dependency from 1.3.12 to 1.5.1 in /amazon-kinesis-client and /amazon-kinesis-client-multilang and aws-java-sdk.version from 1.12.405 to 1.12.668 in /amazon-kinesis-client-multilang

* Upgrade ch.qos.logback:logback-classic dependency from 1.5.1 to 1.3.14 in /amazon-kinesis-client and /amazon-kinesis-client-multilang and upgrade awssdk.version from 2.20.43 to 2.25.3

* upgrade awssdk.version from 2.20.43 to 2.25.3

* Upgrade gsr.version from 1.1.17 to 1.1.19

---------

Co-authored-by: Brendan Lynch <brenplyn@amazon.com>
 2024-03-07 14:37:40 -08:00
kcl-release-automation-bot
b9ec494b02
Preparation for v2.5.5 (#1267) 2024-02-23 14:27:32 -08:00
zachjhum
5f3de14c88
Prevent improper error logging during worker shutdown (#1257) 
* Move throwOnIllegalState call to drain queue method to prevent improper error logging during worker shutdown

* Fix unit tests that expected IllegalStateException thrown

* Changed names of unit tests to reflect new behavior
 2024-02-21 13:19:50 -08:00
lucienlu-aws
b2eb38e510
Add Deletion protection config (#1260) 
* Add deletionProtectionEnabled config
 2024-02-20 14:46:54 -08:00
lucienlu-aws
2d769733fe
Fix issue in configuring metricsEnabledDimensions (#1258) 2024-02-20 13:21:03 -08:00
lucienlu-aws
fb6ab3f0bc
Add snapshot to version (#1259) 2024-02-20 13:20:31 -08:00
kcl-release-automation-bot
a5d1c67660
Preparation for v2.5.4 (#1240) 2023-12-13 13:05:16 -08:00
dependabot[bot]
1727765d87
Bump ch.qos.logback:logback-classic in /amazon-kinesis-client (#1232) 
Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.3.0 to 1.3.12.
- [Commits](https://github.com/qos-ch/logback/compare/v_1.3.0...v_1.3.12)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-04 12:12:36 -08:00
dependabot[bot]
1701eb3763
Bump ch.qos.logback:logback-classic in /amazon-kinesis-client-multilang (#1233) 
Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.3.0 to 1.3.12.
- [Commits](https://github.com/qos-ch/logback/compare/v_1.3.0...v_1.3.12)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-04 12:12:12 -08:00
Aravinda Kidambi Srinivasan
44837b702a
Fix an issue in configuring idleTimeBetweenReadsInMillis in MultiLangDaemon (#1230) 
Fix an issue where the idleTimeBetweenReadInMillis configured
via MultiLangDaemon was not taking effect because it used
the auto-generated setter from Lombok to set the configured value,
while there is a custom setter that must be invoked to set the
value correctly.

There is also a general confusion between using Lombok's setter vs
custom setter in java.

Unifying the approach to use the custom Lombok-fluent-style setter
and deprecating the previously added custom setIdleTimeBetweenReadsInMillis

Correct way to configure idleTimeBetweenReadsInMillis for MultiLang is
to add this in the properties file:
idleTimeBetweenReadsInMillis = 10000 # 10 seconds

Correct way to configure for java:
configsBuilder.retrievalConfig().retrievalSpecificConfig(
    new PollingConfig(streamName, kinesisClient)
        .idleTimeBetweenReadsInMillis(Duration.ofSeconds(10).toMillis())

Issues: #999, #950, #515
 2023-11-22 17:30:41 -08:00
stair
a48f5436ee
Added link to javadoc.io-hosted Javadoc. (#1229) 2023-11-16 17:37:50 -05:00
stair
51a62a559c
Added doc for leases and the lease lifecycle. (#1218) 
* Added doc for leases and the lease lifecycle.

* Documentation: addressed comments for leases.
+ minor code cleanup
* Documentation: language review.
+ decomposed shard sync UML into two separate diagrams (initialization, loop)
* Documentation: language review touch-ups.
 2023-11-10 14:56:15 -05:00