-
Notifications
You must be signed in to change notification settings - Fork 264
A50: Exclude locally-initiated client cancellations from Call Counter (#12923) #558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
f8a754c
11fbd2f
0ec0d2c
b027e8d
ae26586
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -137,7 +137,7 @@ When the `outlier_detection` LB policy receives an address update, it will creat | |
|
|
||
| When the child policy asks for a subchannel, the `outlier_detection` will wrap the subchannel with a wrapper (see [Subchannel Wrapper section](#subchannel-wrapper)). Then, the subchannel wrapper will be added to the list in the map entry for its address, if that map entry exists. If there is no map entry, or if the subchannel is created with multiple addresses, the subchannel will be ignored for outlier detection. If that address is currently ejected, that subchannel wrapper's `eject` method will be called. | ||
|
|
||
| The `outlier_detection` LB policy will provide a picker that delegates to the child policy's picker, and when the request finishes, increment the corresponding counter in the map entry referenced by the subchannel wrapper that was picked. If both the `success_rate_ejection` and `failure_percentage_ejection` fields are unset in the configuration, the picker should not do that counting. | ||
| The `outlier_detection` LB policy will provide a picker that delegates to the child policy's picker, and when the request finishes, increment the corresponding counter in the map entry referenced by the subchannel wrapper that was picked. Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) MUST be excluded from `successCount` and `failureCount`. If both the `success_rate_ejection` and `failure_percentage_ejection` fields are unset in the configuration, the picker should not do that counting. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't do MUST. This document is not using that style.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Referencing I think we could avoid the change in this paragraph. The Call Counter section is the one that looks to be really defining this. Maybe we can make this section more generic so it won't be assumed to be thorough by At that point I think we remove defining |
||
|
|
||
| The `outlier_detection` LB policy will have a timer that triggers on a period determined by the `interval` config option, and does the following: | ||
|
|
||
|
|
@@ -170,7 +170,7 @@ To un-eject an address, set the current ejection timestamp to `null` and call `u | |
|
|
||
| ### Call Counter | ||
|
|
||
| This design is based directly on Envoy's implementation. The object contains two buckets, and each bucket has a number counting successes, and another counting failures. The active bucket is updated each time a call finishes. When the timer triggers, the inactive bucket is zeroed and swapped with the active bucket. Then the inactive bucket contains the number of successes and failures since the last time the timer triggered. Those numbers are used to evaluate the ejection criteria. | ||
| This design is based directly on Envoy's implementation. The object contains two buckets, and each bucket has a number counting successes (`successCount`), and another counting failures (`failureCount`). The active bucket is updated each time a call finishes, excluding locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`). When the timer triggers, the inactive bucket is zeroed and swapped with the active bucket. Then the inactive bucket contains the number of successes and failures since the last time the timer triggered. Those numbers are used to evaluate the ejection criteria. | ||
|
|
||
| ### Subchannel Wrapper | ||
|
|
||
|
|
@@ -282,6 +282,10 @@ Envoy's specification of outlier detection includes the ejection criteria Consec | |
|
|
||
| Envoy defines some errors as "external" and some as "local origin", and their specification of outlier detection allows separate configurations for handling each of them. gRPC does not separate errors that way, so there is no way to split them like that and handle those two categories separately. | ||
|
|
||
| ### Excluding Locally-Initiated Client-Side Cancellations | ||
|
|
||
| Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) are excluded from outlier detection counting (`successCount` and `failureCount`). For locally-initiated cancellations (hedging cancellations of non-winning sibling attempts and application-initiated cancellations), this aligns with Envoy's `resetStream()` behavior and prevents false-positive ejections during hedging. For `DEADLINE_EXCEEDED`, this differs from Envoy (which counts deadline expirations as failures for outlier detection); however, because deadline expiration semantics vary between Envoy and gRPC, and because a deadline expiration cannot be definitively attributed to a server failure rather than a network delay, gRPC treats it as a client-side cancellation. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "semantics vary" is pretty vague. "gRPC can't reliably distinguish between cancellation and deadline exceeded cross-language" seems relevant. |
||
|
|
||
| ### Map Entry Source | ||
|
|
||
| We chose to populate the map entries using the address list passed in from the parent LB policy. As an alternative, we considered populating the map using addresses that the child LB policy uses to create subchannels. These two options would have approximately the same behavior with all common child LB policies, and this choice simplifies management of the lifecycle of map entries. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
application CANCELLED and DEADLINE_EXCEEDED shouldn't be in all-upper-case, as it makes it seem like this is based on the status code. Just describe them. ("application cancellation, and deadline exceeded")