AIP-97 fast-follow: classify node drain and preemption as infra - #24
Closed
1fanwang wants to merge 1 commit into
Closed
AIP-97 fast-follow: classify node drain and preemption as infra#241fanwang wants to merge 1 commit into
1fanwang wants to merge 1 commit into
Conversation
A node drain and a scheduler preemption are infrastructure taking the pod away, but Airflow reported both to the listener as an application failure and refunded nothing. Verified live on a k8s v1.35.0 cluster: both reach phase=Failed with pod.status.reason empty and the container reading only Error/exit 143, so neither the pod reason nor the container reason can see them. The signal lives in pod.status.conditions[type=DisruptionTarget], which the control plane sets before the delete and which survives onto the terminal object. Read that condition in collect_pod_failure_details and check it first in classify_pod_failure. Gated on status "True", matching Kubernetes' own podFailurePolicy matcher, since the writers update the condition in place. This closes the gap the parent PR disclosed as out of scope. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Owner
Author
|
Folded into the consolidated AIP-97 PR as The runtime cost is +42/-1 across two files, and the AIP already described a node drain and a preemption as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fast-follow on #66405 (stacked on its head). Kept out of that PR to hold its scope to the general mechanism.
Why
A node drain and a scheduler preemption are infrastructure taking the pod away, but
classify_pod_failure()reports both asapplication, so neither gets the refund AIP-97 exists to give. Node drain covers cordon/drain and autoscaler scale-down, so this is the common case in a real cluster, not a corner.Verified on a live kind cluster (k8s v1.35.0) by recording every watch event: both disruptions reach
phase=Failedwithpod.status.reasonempty and the container reading onlyError/exit 143, which is indistinguishable from an application crash. TheDELETEDevent also arrives atphase=Failed, so the watcher's deleted-pod branch never fires either.The signal is
pod.status.conditions[type=DisruptionTarget]. The control plane sets it before the delete and it survives onto the terminal object, which is exactly where the classifier can read it.What
collect_pod_failure_details()records the condition's reason, andclassify_pod_failure()checks it before the reason sets. Gated onstatus: "True", matching Kubernetes' own podFailurePolicy matcher, since the writers update the condition in place. The four reasons are pinned to their Go definitions, same convention as the existing set.Testing
Red/green against the pre-fix source, then the live cluster runs the fix is built from.
Red to green
Live watch events, kind v1.35.0
Node drain, via
kubectl taint node <node> aip97=evict:NoExecute:Scheduler preemption, via a higher-priority pod that cannot otherwise fit:
status.reasonis empty throughout and the container only ever saysError/143, so the condition is the sole signal. The end-to-end test builds this exact object.ruffandmypyclean.