Update kcl-configurations.md
Add two KCL configurations - WorkerMetricsTableConfig, CoordinatorStateTableConfig - to let users know how to set the custom table name.
This commit is contained in:
Minu Hong
GitHub
parent
d708ecc0e2
commit
c560294612
1 changed files with 57 additions and 50 deletions
|
|
@ -7,7 +7,7 @@ You can set configuration properties to customize Kinesis Client Library's funct
|
|||
|
||||
## List of KCL configuration properties
|
||||
| Configuration property | Configuration class | Description | Default value |
|
||||
|--------------------------------------------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------:|
|
||||
| ------------------------------------------------ | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------: |
|
||||
| applicationName | ConfigsBuilder | The name for this the KCL application. Used as the default for the tableName and consumerName. | N/A |
|
||||
| tableName | ConfigsBuilder | Allows overriding the table name used for the Amazon DynamoDB lease table. | N/A |
|
||||
| streamName | ConfigsBuilder | The name of the stream that this application processes records from. | N/A |
|
||||
|
|
@ -29,6 +29,7 @@ You can set configuration properties to customize Kinesis Client Library's funct
|
|||
| maxThroughputPerHostKBps | LeaseManagementConfig | Amount of the maximum throughput to assign to a worker during the lease assignment. This is a new configuration introduced in KCL 3.x. | Unlimited |
|
||||
| isGracefulLeaseHandoffEnabled | LeaseManagementConfig | Controls the behavior of lease handoff between workers. When set to true, KCL will attempt to gracefully transfer leases by allowing the shard's RecordProcessor sufficient time to complete processing before handing off the lease to another worker. This can help ensure data integrity and smooth transitions but may increase handoff time. When set to false, the lease will be handed off immediately without waiting for the RecordProcessor to shut down gracefully. This can lead to faster handoffs but may risk incomplete processing. Note: Checkpointing must be implemented inside the shutdownRequested() method of the RecordProcessor to get benefited from the graceful lease handoff feature. This is a new configuration introduced in KCL 3.x. | TRUE |
|
||||
| gracefulLeaseHandoffTimeoutMillis | LeaseManagementConfig | Specifies the minimum time (in milliseconds) to wait for the current shard's RecordProcessor to gracefully shut down before forcefully transferring the lease to the next owner. If your processRecords method typically runs longer than the default value, consider increasing this setting. This ensures the RecordProcessor has sufficient time to complete its processing before the lease transfer occurs. This is a new configuration introduced in KCL 3.x. | 30000 (30 seconds) |
|
||||
| WorkerMetricsTableConfig | LeaseManagementConfig | This configuration determines the DynamoDB table settings such as table name, billing mode, provisioned capacity, PITR, deletion protection, and tags for the worker metrics table. | By default, KCL creates a worker metrics table named `<KCLApplicationName>-WorkerMetricStats` with on-demand billing mode. |
|
||||
| maxRecords | PollingConfig | Allows setting the maximum number of records that Kinesis returns. | 10000 |
|
||||
| retryGetRecordsInSeconds | PollingConfig | Configures the delay between GetRecords attempts for failures. | none |
|
||||
| maxGetRecordsThreadPool | PollingConfig | The thread pool size used for GetRecords. | none |
|
||||
|
|
@ -38,6 +39,7 @@ You can set configuration properties to customize Kinesis Client Library's funct
|
|||
| skipShardSyncAtWorkerInitializationIfLeasesExist | CoordinatorConfig | Disable synchronizing shard data if the lease table contains existing leases. | FALSE |
|
||||
| shardPrioritization | CoordinatorConfig | Which shard prioritization to use. | NoOpShardPrioritization |
|
||||
| ClientVersionConfig | CoordinatorConfig | Determines which KCL version compatibility mode the application will run in. This configuration is only for the migration from previous KCL versions. When migrating to 3.x, you need to set this configuration to CLIENT_VERSION_CONFIG_COMPATIBLE_WITH_2x. You can remove this configuration when you complete the migration. | CLIENT_VERSION_CONFIG_3X |
|
||||
| CoordinatorStateTableConfig | CoordinatorConfig | This configuration determines the DynamoDB table settings such as table name, billing mode, provisioned capacity, PITR, deletion protection, and tags for the coordinator state table. | By default, KCL creates a worker metrics table named `<KCLApplicationName>-CoordinatorState` with on-demand billing mode. |
|
||||
| taskBackoffTimeMillis | LifecycleConfig | The time to wait to retry failed KCL tasks. The unit is milliseconds. | 500 (0.5 seconds) |
|
||||
| logWarningForTaskAfterMillis | LifecycleConfig | How long to wait before a warning is logged if a task hasn't completed. | none |
|
||||
| listShardsBackoffTimeInMillis | RetrievalConfig | The number of milliseconds to wait between calls to ListShards when failures occur. The unit is milliseconds. | 1500 (1.5 seconds) |
|
||||
|
|
@ -47,10 +49,13 @@ You can set configuration properties to customize Kinesis Client Library's funct
|
|||
| metricsLevel | MetricsConfig | Specifies the granularity level of CloudWatch metrics to be enabled and published. Possible values: NONE, SUMMARY, DETAILED. | MetricsLevel.DETAILED |
|
||||
| metricsEnabledDimensions | MetricsConfig | Controls allowed dimensions for CloudWatch Metrics. | All dimensions |
|
||||
|
||||
|
||||
|
||||
|
||||
## New configurations in KCL 3.x
|
||||
The following configuration properties are newly added in KCL 3.x:
|
||||
| Configuration property | Configuration class | Description | Default value |
|
||||
|---|---|---|---:|
|
||||
| --------------------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------: |
|
||||
| reBalanceThresholdPercentage | LeaseManagementConfig | A percentage value that determines when the load balancing algorithm should consider reassigning shards among workers. This is a new configuration introduced in KCL 3.x. | 10 |
|
||||
| dampeningPercentage | LeaseManagementConfig | A percentage value that is used to dampen the amount of load that will be moved from the overloaded worker in a single rebalance operation. This is a new configuration introduced in KCL 3.x. | 60 |
|
||||
| allowThroughputOvershoot | LeaseManagementConfig | Determines whether additional lease still needs to be taken from the overloaded worker even if it causes total amount of lease throughput taken to exceed the desired throughput amount. This is a new configuration introduced in KCL 3.x. | TRUE |
|
||||
|
|
@ -58,6 +63,8 @@ The following configuration properties are newly added in KCL 3.x:
|
|||
| maxThroughputPerHostKBps | LeaseManagementConfig | Amount of the maximum throughput to assign to a worker during the lease assignment. This is a new configuration introduced in KCL 3.x. | Unlimited |
|
||||
| isGracefulLeaseHandoffEnabled | LeaseManagementConfig | Controls the behavior of lease handoff between workers. When set to true, KCL will attempt to gracefully transfer leases by allowing the shard's RecordProcessor sufficient time to complete processing before handing off the lease to another worker. This can help ensure data integrity and smooth transitions but may increase handoff time. When set to false, the lease will be handed off immediately without waiting for the RecordProcessor to shut down gracefully. This can lead to faster handoffs but may risk incomplete processing. Note: Checkpointing must be implemented inside the shutdownRequested() method of the RecordProcessor to get benefited from the graceful lease handoff feature. This is a new configuration introduced in KCL 3.x. | TRUE |
|
||||
| gracefulLeaseHandoffTimeoutMillis | LeaseManagementConfig | Specifies the minimum time (in milliseconds) to wait for the current shard's RecordProcessor to gracefully shut down before forcefully transferring the lease to the next owner. If your processRecords method typically runs longer than the default value, consider increasing this setting. This ensures the RecordProcessor has sufficient time to complete its processing before the lease transfer occurs. This is a new configuration introduced in KCL 3.x. | 30000 (30 seconds) |
|
||||
| WorkerMetricsTableConfig | LeaseManagementConfig | This configuration determines the DynamoDB table settings such as table name, billing mode, provisioned capacity, PITR, deletion protection, and tags for the worker metrics table. | By default, KCL creates a worker metrics table named `<KCLApplicationName>-WorkerMetricStats` with on-demand billing mode. |
|
||||
| CoordinatorStateTableConfig | CoordinatorConfig | This configuration determines the DynamoDB table settings such as table name, billing mode, provisioned capacity, PITR, deletion protection, and tags for the coordinator state table. | By default, KCL creates a worker metrics table named `<KCLApplicationName>-CoordinatorState` with on-demand billing mode. |
|
||||
|
||||
|
||||
## Discontinued configuration properties in KCL 3.x
|
||||
|
|
|
|||
Loading…
Reference in a new issue