From 7bc701fa321935a79f7b14311eb36da4295bd50b Mon Sep 17 00:00:00 2001 From: Yatin Date: Mon, 19 Oct 2020 16:38:40 -0700 Subject: [PATCH 1/2] KCL release v2.3.1. --- CHANGELOG.md | 11 +++- README.md | 58 +++---------------- amazon-kinesis-client-multilang/pom.xml | 2 +- amazon-kinesis-client/pom.xml | 2 +- .../kinesis/retrieval/RetrievalConfig.java | 2 +- pom.xml | 2 +- 6 files changed, 22 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e3e271b3..2ba9c029 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,16 @@ # Changelog -### Latest Release (2.3.0 - August 17, 2020) +### Latest Release 2.3.1 (October 20, 2020) +[Milestone#53](https://github.com/awslabs/amazon-kinesis-client/milestone/53) +* Introducing support for processing multiple kinesis data streams with the same KCL 2.x for java consumer application + * To build a consumer application that can process multiple streams at the same time, you must implement a new + interface called MultistreamTracker (https://github.com/awslabs/amazon-kinesis-client/blob/0c5042dadf794fe988438436252a5a8fe70b6b0b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/MultiStreamTracker.java) + * MultistreamTracker will also publish various metrics around the current active streams being processed, the number + of streams which are deleted at this time period or are pending deletion. + + +### Release 2.3.0 (August 17, 2020) * [Milestone#52](https://github.com/awslabs/amazon-kinesis-client/milestones/52) * Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding. diff --git a/README.md b/README.md index 12a00f04..31b6c7b5 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ The recommended way to use the KCL for Java is to consume it from Maven. software.amazon.kinesis amazon-kinesis-client - 2.3.0 + 2.3.1 ``` @@ -63,56 +63,14 @@ The recommended way to use the KCL for Java is to consume it from Maven. ## Release Notes -### Latest Release (2.3.0 - August 17, 2020) +### Latest Release 2.3.1 (October 20, 2020) +[Milestone#53](https://github.com/awslabs/amazon-kinesis-client/milestone/53) +* Introducing support for processing multiple kinesis data streams with the same KCL 2.x for java consumer application + * To build a consumer application that can process multiple streams at the same time, you must implement a new + interface called MultistreamTracker (https://github.com/awslabs/amazon-kinesis-client/blob/0c5042dadf794fe988438436252a5a8fe70b6b0b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/MultiStreamTracker.java) -* [Milestone#52](https://github.com/awslabs/amazon-kinesis-client/milestones/52) - -* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding. -* When bootstrapping an empty lease table, KCL utilizes the `ListShard` API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the `ShardFilter` parameter. The `ShardFilter` parameter enables you to filter out the response of the `ListShards` API, using the `Type` parameter. KCL uses the `Type` filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases. - * Currently, the following shard filters are supported: - * `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`. - * `AT_LATEST` - the response includes only the currently open shards of the data stream. - * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open. - * `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `RetrievalConfig#initialPositionInStreamExtended`. - * For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html). - -* Introducing support for the `ChildShards` response of the `GetRecords` and the `SubscribeToShard` APIs to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing. - * For shared throughout consumer applications, this uses the `ChildShards` response of the `GetRecords` API. For dedicated throughput (enhanced fan-out) consumer applications, this uses the `ChildShards` response of the `SubscribeToShard` API. - * For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html), [SubscribeToShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html), and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html). - -* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. `PeriodicShardSyncManager` is the new component that is responsible for running periodic lease/shard scans. - * New configuration options are available to configure `PeriodicShardSyncManager` in `LeaseManagementConfig` - - | Name | Default | Description | - | ----------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | leasesRecoveryAuditorExecutionFrequencyMillis | 120000 (2 minutes) | Frequency (in millis) of the auditor job to scan for partial leases in the lease table. If the auditor detects any hole in the leases for a stream, then it would trigger shard sync based on leasesRecoveryAuditorInconsistencyConfidenceThreshold. | - | leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3 | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync | - - * New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`: - - | Name | Description | - | --------------------------- | ------------------------------------------------------ | - | NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. | - | NumStreamsToSync | Number of streams which underwent a full shard sync. | - -* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation. - * New configuration options are available to configure `LeaseCleanupManager`. - - | Name | Default | Description | - | ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- | - | leaseCleanupIntervalMillis | 1 minute | Interval at which to run lease cleanup thread. | - | completedLeaseCleanupIntervalMillis | 5 minutes | Interval at which to check if a lease is completed or not. | - | garbageLeaseCleanupIntervalMillis | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. | - -* Introducing _experimental_ support for multistreaming, allowing a single KCL application to multiplex processing multiple streams. - * New configuration options are available to enable multistreaming in `RetrievalConfig#appStreamTracker`. - -* Fixing a bug in `PrefetchRecordsPublisher` restarting while it was already running. -* Including an optimization to `HierarchicalShardSyncer` to only create leases for one layer of shards. -* Adding support to prepare and commit lease checkpoints with arbitrary bytes. - * This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size ([currently 400 KB as of release](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html)), and can be used for recovery by passing a serialized byte buffer to `RecordProcessorCheckpointer#prepareCheckpoint` and `RecordProcessorCheckpointer#checkpoint`. -* Upgrading version of AWS SDK to 2.14.0. -* [#725](https://github.com/awslabs/amazon-kinesis-client/pull/725) Allowing KCL to consider lease tables in `UPDATING` healthy. + * MultistreamTracker will also publish various metrics around the current active streams being processed, the number + of streams which are deleted at this time period or are pending deletion. ### For remaining release notes check **[CHANGELOG.md][changelog-md]**. diff --git a/amazon-kinesis-client-multilang/pom.xml b/amazon-kinesis-client-multilang/pom.xml index c76b95c5..8b01925a 100644 --- a/amazon-kinesis-client-multilang/pom.xml +++ b/amazon-kinesis-client-multilang/pom.xml @@ -21,7 +21,7 @@ amazon-kinesis-client-pom software.amazon.kinesis - 2.3.0 + 2.3.1 4.0.0 diff --git a/amazon-kinesis-client/pom.xml b/amazon-kinesis-client/pom.xml index 55fd63b4..c01d6160 100644 --- a/amazon-kinesis-client/pom.xml +++ b/amazon-kinesis-client/pom.xml @@ -22,7 +22,7 @@ software.amazon.kinesis amazon-kinesis-client-pom - 2.3.0 + 2.3.1 amazon-kinesis-client diff --git a/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/RetrievalConfig.java b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/RetrievalConfig.java index e3ef9d63..9ab4ca9e 100644 --- a/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/RetrievalConfig.java +++ b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/RetrievalConfig.java @@ -45,7 +45,7 @@ public class RetrievalConfig { */ public static final String KINESIS_CLIENT_LIB_USER_AGENT = "amazon-kinesis-client-library-java"; - public static final String KINESIS_CLIENT_LIB_USER_AGENT_VERSION = "2.3.0"; + public static final String KINESIS_CLIENT_LIB_USER_AGENT_VERSION = "2.3.1"; /** * Client used to make calls to Kinesis for records retrieval diff --git a/pom.xml b/pom.xml index f8bfd316..4883d75e 100644 --- a/pom.xml +++ b/pom.xml @@ -22,7 +22,7 @@ amazon-kinesis-client-pom pom Amazon Kinesis Client Library - 2.3.0 + 2.3.1 The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data from Amazon Kinesis. From d055a35269dc6a68be0b3226495129a189703d00 Mon Sep 17 00:00:00 2001 From: Yatin Date: Mon, 19 Oct 2020 18:48:33 -0700 Subject: [PATCH 2/2] KCL release v2.3.1. --- README.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/README.md b/README.md index 31b6c7b5..77523fb6 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,56 @@ The recommended way to use the KCL for Java is to consume it from Maven. * MultistreamTracker will also publish various metrics around the current active streams being processed, the number of streams which are deleted at this time period or are pending deletion. +### Release 2.3.0 (August 17, 2020) +* [Milestone#52](https://github.com/awslabs/amazon-kinesis-client/milestones/52) + +* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding. +* When bootstrapping an empty lease table, KCL utilizes the `ListShard` API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the `ShardFilter` parameter. The `ShardFilter` parameter enables you to filter out the response of the `ListShards` API, using the `Type` parameter. KCL uses the `Type` filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases. + * Currently, the following shard filters are supported: + * `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`. + * `AT_LATEST` - the response includes only the currently open shards of the data stream. + * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open. + * `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `RetrievalConfig#initialPositionInStreamExtended`. + * For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html). + +* Introducing support for the `ChildShards` response of the `GetRecords` and the `SubscribeToShard` APIs to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing. + * For shared throughout consumer applications, this uses the `ChildShards` response of the `GetRecords` API. For dedicated throughput (enhanced fan-out) consumer applications, this uses the `ChildShards` response of the `SubscribeToShard` API. + * For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html), [SubscribeToShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html), and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html). + +* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. `PeriodicShardSyncManager` is the new component that is responsible for running periodic lease/shard scans. + * New configuration options are available to configure `PeriodicShardSyncManager` in `LeaseManagementConfig` + + | Name | Default | Description | + | ----------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | leasesRecoveryAuditorExecutionFrequencyMillis | 120000 (2 minutes) | Frequency (in millis) of the auditor job to scan for partial leases in the lease table. If the auditor detects any hole in the leases for a stream, then it would trigger shard sync based on leasesRecoveryAuditorInconsistencyConfidenceThreshold. | + | leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3 | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync | + + * New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`: + + | Name | Description | + | --------------------------- | ------------------------------------------------------ | + | NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. | + | NumStreamsToSync | Number of streams which underwent a full shard sync. | + +* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation. + * New configuration options are available to configure `LeaseCleanupManager`. + + | Name | Default | Description | + | ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- | + | leaseCleanupIntervalMillis | 1 minute | Interval at which to run lease cleanup thread. | + | completedLeaseCleanupIntervalMillis | 5 minutes | Interval at which to check if a lease is completed or not. | + | garbageLeaseCleanupIntervalMillis | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. | + +* Introducing _experimental_ support for multistreaming, allowing a single KCL application to multiplex processing multiple streams. + * New configuration options are available to enable multistreaming in `RetrievalConfig#appStreamTracker`. + +* Fixing a bug in `PrefetchRecordsPublisher` restarting while it was already running. +* Including an optimization to `HierarchicalShardSyncer` to only create leases for one layer of shards. +* Adding support to prepare and commit lease checkpoints with arbitrary bytes. + * This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size ([currently 400 KB as of release](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html)), and can be used for recovery by passing a serialized byte buffer to `RecordProcessorCheckpointer#prepareCheckpoint` and `RecordProcessorCheckpointer#checkpoint`. +* Upgrading version of AWS SDK to 2.14.0. +* [#725](https://github.com/awslabs/amazon-kinesis-client/pull/725) Allowing KCL to consider lease tables in `UPDATING` healthy. + ### For remaining release notes check **[CHANGELOG.md][changelog-md]**. [kinesis]: http://aws.amazon.com/kinesis