IGNITE-28305 Add backpressure for partition operations#7950
Open
EgorKuts wants to merge 1 commit intoapache:mainfrom
Open
IGNITE-28305 Add backpressure for partition operations#7950EgorKuts wants to merge 1 commit intoapache:mainfrom
EgorKuts wants to merge 1 commit intoapache:mainfrom
Conversation
EgorKuts
force-pushed
the
ignite-28305
branch
7 times, most recently
from
9140021 to
4fd15ab
Compare
| * When positive, {@link #tryAcquire()} returns {@code false} once the limit is reached and the caller should reject the request. | ||
| * A permit must be released via {@link #release()} upon operation completes. | ||
| */ | ||
| public class PartitionOperationInFlightLimiter { |
Contributor
There was a problem hiding this comment.
Suggested change
| public class PartitionOperationInFlightLimiter { | |
| public class PartitionOperationInflightLimiter { |
| return result; | ||
| /** Constructor. */ | ||
| public ReplicaOverloadedException() { | ||
| super(GROUP_OVERLOADED_ERR, "Node is overloaded: max in-flight partition operations limit reached."); |
Contributor
There was a problem hiding this comment.
A separate error code is needed: REPLICA_OVERLOADED_ERR
| if (ClientOp.isPartitionOperation(opCode)) { | ||
| long requestId0 = requestId; | ||
| int opCode0 = opCode; | ||
| if (!partitionOperationInFlightLimiter.tryAcquire()) { |
Contributor
There was a problem hiding this comment.
This approach doesn't look working to me.
- It make single get operation and a write batch of thousand keys equal in terms of backpressure.
- It's not clear how a used should choose the proper "per-core" limit value. Using a small value will throttle too much, and big value basically disables throttling.
- It is disabled by default, which means a cluster is not protected from overloading by this mechanism.
We already have natural backpressure in the form of max lock table size.
Probably this is enough.
I would start from writing tests which actually overload the cluster and figure out how to avoid this by introducing new natual backpressure metrics.
For example, a number of active transactions per node.
ptupitsyn
changed the title
ptupitsyn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change, a node could accept an unbounded number of partition operations, which could lead to the following problems:
Long-queued operations no longer make sense to the client by the time they are executed, forcing the node to perform useless work before it can start handling new requests.
That useless work still consumes node resources (CPU, memory, threads).
Eventually this can lead to OOM.
To prevent such scenarios, this change introduces a node-level semaphore shared across the replica manager and thin-client connector that limits the total number of concurrent partition operations to
maxInFlightPartitionOperationsPerCore * availableProcessors
When the limit is reached, new requests are rejected immediately with ReplicaOverloadedException so clients can back off, while already accepted operations complete uninterrupted.
The limit is disabled by default
(maxInFlightPartitionOperationsPerCore = 0)
Operator can start tuning this around 512 per core and adjust based on observed heap usage and rejection rate under peak load.
https://issues.apache.org/jira/browse/IGNITE-28305