Skip to content

AIP-97 fast-follow: classify node drain and preemption as infra - #24

Closed
1fanwang wants to merge 1 commit into
1fanwang/aip97-failure-detailsfrom
1fanwang/aip97-disruption-condition
Closed

AIP-97 fast-follow: classify node drain and preemption as infra#24
1fanwang wants to merge 1 commit into
1fanwang/aip97-failure-detailsfrom
1fanwang/aip97-disruption-condition

Conversation

@1fanwang

Copy link
Copy Markdown
Owner

Fast-follow on #66405 (stacked on its head). Kept out of that PR to hold its scope to the general mechanism.

Why

A node drain and a scheduler preemption are infrastructure taking the pod away, but classify_pod_failure() reports both as application, so neither gets the refund AIP-97 exists to give. Node drain covers cordon/drain and autoscaler scale-down, so this is the common case in a real cluster, not a corner.

Verified on a live kind cluster (k8s v1.35.0) by recording every watch event: both disruptions reach phase=Failed with pod.status.reason empty and the container reading only Error/exit 143, which is indistinguishable from an application crash. The DELETED event also arrives at phase=Failed, so the watcher's deleted-pod branch never fires either.

The signal is pod.status.conditions[type=DisruptionTarget]. The control plane sets it before the delete and it survives onto the terminal object, which is exactly where the classifier can read it.

What

collect_pod_failure_details() records the condition's reason, and classify_pod_failure() checks it before the reason sets. Gated on status: "True", matching Kubernetes' own podFailurePolicy matcher, since the writers update the condition in place. The four reasons are pinned to their Go definitions, same convention as the existing set.

Testing

Red/green against the pre-fix source, then the live cluster runs the fix is built from.

Red to green
$ git stash push -- providers/cncf/kubernetes/src/   # tests against pre-fix source
$ pytest .../test_classify_pod_failure.py -q -p no:randomly
8 failed, 16 passed

$ git stash pop
$ pytest .../test_classify_pod_failure.py -q -p no:randomly
24 passed
Live watch events, kind v1.35.0

Node drain, via kubectl taint node <node> aip97=evict:NoExecute:

  12.4s  type=MODIFIED  phase=Running  deletionTS=-    status.reason=None  DisruptionTarget=True/DeletionByTaintManager  container=-
  12.4s  type=MODIFIED  phase=Running  deletionTS=SET  status.reason=None  DisruptionTarget=True/DeletionByTaintManager  container=-
  17.8s  type=MODIFIED  phase=Failed   deletionTS=SET  status.reason=None  DisruptionTarget=True/DeletionByTaintManager  container=term:Error:exit=143
  18.1s  type=DELETED   phase=Failed   deletionTS=SET  status.reason=None  DisruptionTarget=True/DeletionByTaintManager  container=term:Error:exit=143

Scheduler preemption, via a higher-priority pod that cannot otherwise fit:

  12.5s  type=MODIFIED  phase=Running  deletionTS=-    status.reason=None  DisruptionTarget=True/PreemptionByScheduler  container=-
  12.5s  type=MODIFIED  phase=Running  deletionTS=SET  status.reason=None  DisruptionTarget=True/PreemptionByScheduler  container=-
  17.7s  type=MODIFIED  phase=Failed   deletionTS=SET  status.reason=None  DisruptionTarget=True/PreemptionByScheduler  container=term:Error:exit=143
  18.2s  type=DELETED   phase=Failed   deletionTS=SET  status.reason=None  DisruptionTarget=True/PreemptionByScheduler  container=term:Error:exit=143

status.reason is empty throughout and the container only ever says Error/143, so the condition is the sole signal. The end-to-end test builds this exact object.

ruff and mypy clean.

A node drain and a scheduler preemption are infrastructure taking the pod
away, but Airflow reported both to the listener as an application failure and
refunded nothing. Verified live on a k8s v1.35.0 cluster: both reach
phase=Failed with pod.status.reason empty and the container reading only
Error/exit 143, so neither the pod reason nor the container reason can see
them. The signal lives in pod.status.conditions[type=DisruptionTarget], which
the control plane sets before the delete and which survives onto the terminal
object.

Read that condition in collect_pod_failure_details and check it first in
classify_pod_failure. Gated on status "True", matching Kubernetes' own
podFailurePolicy matcher, since the writers update the condition in place.

This closes the gap the parent PR disclosed as out of scope.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang

Copy link
Copy Markdown
Owner Author

Folded into the consolidated AIP-97 PR as 8ae4f32a3a: apache#66405

The runtime cost is +42/-1 across two files, and the AIP already described a node drain and a preemption as infra, so shipping the reference classifier without the condition read left the text and the code disagreeing. Keeping the branch for the live evidence.

@1fanwang 1fanwang closed this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant