AIP-97 POC (open q): drop infra_reason column, deliver the token via error - #22
Open
1fanwang wants to merge 4 commits into
Open
AIP-97 POC (open q): drop infra_reason column, deliver the token via error#221fanwang wants to merge 4 commits into
1fanwang wants to merge 4 commits into
Conversation
…column on_task_instance_failed gains an optional failure_kind argument, a TaskFailureKind str enum (infra / application / timeout / manual) in airflow_shared.state. pluggy dispatches by parameter name, so a listener that does not declare it keeps working. The worker populates application or timeout; a manual mark-failed populates manual. The executor's reason token persists on the task instance as a new nullable infra_reason column. Foundation for AIP-97; the Kubernetes classifier (infra) and the opt-in retry refund follow as separate PRs. Signed-off-by: 1fanwang <1fannnw@gmail.com>
…he scheduler The Kubernetes executor's classify_pod_failure() maps a node-level disruption (Evicted, Preempting, DisruptionTarget, node shutdown) or a pod removed while its task is running (PodDeleted — drain, preemption, spot reclaim, force-delete) to failure_kind=infra; a container that ended on its own (an app crash, an OOM against its own limit) to application. The classification rides a side channel on BaseExecutor that the scheduler reads once and passes to the listener with the executor's reason token. An unclassified death stays None, so the scheduler stays conservative and never guesses infra. Builds on the failure_kind foundation; the opt-in retry refund follows. Signed-off-by: 1fanwang <1fannnw@gmail.com>
…-in) When failure_kind is infra, [core] infra_failure_refund_retries bumps max_tries so the attempt is given back instead of drawing on the user's retries, bounded by [core] max_infra_refunds. Off by default; application, timeout, and manual failures never refund. This extends the Kubernetes executor's existing behavior — a pod that dies before the task starts is already requeued without consuming a retry — to the mid-execution case, now that the failure is classified. Completes the AIP-97 foundation stack. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Remove the infra_reason column + migration. The executor still produces the (failure_kind, reason) token; the scheduler hands the token to the listener as the failure's error instead of persisting it, so an infra failure that left no worker error now surfaces e.g. error='Evicted' with failure_kind=infra. No schema change, no new hook argument. Signed-off-by: 1fanwang <1fannnw@gmail.com>
1fanwang
force-pushed
the
1fanwang/aip97-failure-details
branch
from
July 27, 2026 23:54
c822f45 to
aa82a98
Compare
1fanwang
force-pushed
the
1fanwang/aip97-failure-details
branch
3 times, most recently
from
July 28, 2026 08:35
4a545fd to
0b92f96
Compare
1fanwang
force-pushed
the
1fanwang/aip97-failure-details
branch
3 times, most recently
from
July 29, 2026 05:45
ffc71b0 to
188bb01
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem & Solution Overview
Open-question POC: after dropping the
infra_reasoncolumn, how should the executor's reason token reach the listener?This variant folds the token into the existing
errorargument. An infra failure with no worker error reaches the listener as, for example,error="Evicted"withfailure_kind=infra. That keeps the hook surface unchanged.Trade-off:
errorbecomes a short token for infra failures but a full exception for application failures, so consumers cannot treat it as one uniform shape.Sibling POC, dedicated
reasonargument: #23Both variants are alternatives to the column design in apache#66405.
Testing Done
Shares the verified no-column core with the
reasonPOC: real DB, real registered listener, real scheduler failure path, noinfra_reasoncolumn, transient token delivery, and the retry refunded. The only difference is that this variant sends the token througherrorinstead of a separate argument.