Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions A50-xds-outlier-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ When the `outlier_detection` LB policy receives an address update, it will creat

When the child policy asks for a subchannel, the `outlier_detection` will wrap the subchannel with a wrapper (see [Subchannel Wrapper section](#subchannel-wrapper)). Then, the subchannel wrapper will be added to the list in the map entry for its address, if that map entry exists. If there is no map entry, or if the subchannel is created with multiple addresses, the subchannel will be ignored for outlier detection. If that address is currently ejected, that subchannel wrapper's `eject` method will be called.

The `outlier_detection` LB policy will provide a picker that delegates to the child policy's picker, and when the request finishes, increment the corresponding counter in the map entry referenced by the subchannel wrapper that was picked. If both the `success_rate_ejection` and `failure_percentage_ejection` fields are unset in the configuration, the picker should not do that counting.
The `outlier_detection` LB policy will provide a picker that delegates to the child policy's picker, and when the request finishes, increment the corresponding counter in the map entry referenced by the subchannel wrapper that was picked. Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) MUST be excluded from `successCount` and `failureCount`. If both the `success_rate_ejection` and `failure_percentage_ejection` fields are unset in the configuration, the picker should not do that counting.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

application CANCELLED and DEADLINE_EXCEEDED shouldn't be in all-upper-case, as it makes it seem like this is based on the status code. Just describe them. ("application cancellation, and deadline exceeded")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't do MUST. This document is not using that style.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referencing successCount and failureCount before they are defined is confusing.

I think we could avoid the change in this paragraph. The Call Counter section is the one that looks to be really defining this. Maybe we can make this section more generic so it won't be assumed to be thorough by s/increment the corresponding counter/updates the counter/. Michael may have a preference.

At that point I think we remove defining successCount and failureCount as variables, and remove the reference to the variable names in the rationale.


The `outlier_detection` LB policy will have a timer that triggers on a period determined by the `interval` config option, and does the following:

Expand Down Expand Up @@ -170,7 +170,7 @@ To un-eject an address, set the current ejection timestamp to `null` and call `u

### Call Counter

This design is based directly on Envoy's implementation. The object contains two buckets, and each bucket has a number counting successes, and another counting failures. The active bucket is updated each time a call finishes. When the timer triggers, the inactive bucket is zeroed and swapped with the active bucket. Then the inactive bucket contains the number of successes and failures since the last time the timer triggered. Those numbers are used to evaluate the ejection criteria.
This design is based directly on Envoy's implementation. The object contains two buckets, and each bucket has a number counting successes (`successCount`), and another counting failures (`failureCount`). The active bucket is updated each time a call finishes, excluding locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`). When the timer triggers, the inactive bucket is zeroed and swapped with the active bucket. Then the inactive bucket contains the number of successes and failures since the last time the timer triggered. Those numbers are used to evaluate the ejection criteria.

### Subchannel Wrapper

Expand Down Expand Up @@ -282,6 +282,10 @@ Envoy's specification of outlier detection includes the ejection criteria Consec

Envoy defines some errors as "external" and some as "local origin", and their specification of outlier detection allows separate configurations for handling each of them. gRPC does not separate errors that way, so there is no way to split them like that and handle those two categories separately.

### Excluding Locally-Initiated Client-Side Cancellations

Locally-initiated client-side cancellations (hedging cancellations, application `CANCELLED`, and `DEADLINE_EXCEEDED`) are excluded from outlier detection counting (`successCount` and `failureCount`). For locally-initiated cancellations (hedging cancellations of non-winning sibling attempts and application-initiated cancellations), this aligns with Envoy's `resetStream()` behavior and prevents false-positive ejections during hedging. For `DEADLINE_EXCEEDED`, this differs from Envoy (which counts deadline expirations as failures for outlier detection); however, because deadline expiration semantics vary between Envoy and gRPC, and because a deadline expiration cannot be definitively attributed to a server failure rather than a network delay, gRPC treats it as a client-side cancellation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"semantics vary" is pretty vague. "gRPC can't reliably distinguish between cancellation and deadline exceeded cross-language" seems relevant.


### Map Entry Source

We chose to populate the map entries using the address list passed in from the parent LB policy. As an alternative, we considered populating the map using addresses that the child LB policy uses to create subchannels. These two options would have approximately the same behavior with all common child LB policies, and this choice simplifies management of the lifecycle of map entries.
Expand Down