AIP-97 done-criterion #2: Kubernetes bridge, wired end-to-end (app-OOM excluded, infra refunded)

[1fanwang]

Superseded by #10 — the whole AIP-97 design now lives in #10 as a single clean diff off the apache root (no stack, no add-then-remove churn). This PR is one step of the original incremental stack; kept for history. Review #10.

---

Why

Stacked on #6 (Pillar 2). That PR classifies infra at the scheduler from the state-mismatch signal — enough to tell a self-reporting app failure from a silent death, but not the precise cause. Only the executor holds that: a pod Evicted by the node vs a container OOMKilled against its own limit. This is done-criterion #2 — the Kubernetes bridge, wired end to end.

What

• to_task_failure_info() maps the executor's collect_pod_failure_details() output to a core TaskFailureInfo: a node-level disruption (Evicted/Preempting/NodeShutdown/DisruptionTarget) is infra; a container that ended on its own — an app crash or an OOMKilled against its own limit — is user, so an app OOM earns no infra refund.
• Wiring: the KubernetesExecutor classifies a failed pod in _change_state and stashes the TaskFailureInfo on BaseExecutor.task_failure_info; the scheduler reads it once via get_task_failure_info() and prefers it over the generic infra tag. So the listener and the retry decision get the real pod reason.

Testing Done

Pre/post A/B on a real kind cluster — same deployment, flag off vs on, all 6 modes. Airflow deployed in-cluster (Postgres + api-server + dag-processor + scheduler as pods from this source), KubernetesExecutor launching real task pods, max_infra_refunds=3, a recording listener in the scheduler pod. Every mode run twice; only [core] infra_failure_refund_retries differs between the sweeps.

mode	real failure	RED (flag off)	GREEN (flag on)	Δ
k8s_appbug	task raises ValueError	try=2 max_tries=1	try=2 max_tries=1	invariant
k8s_oom	OOMKilled vs own 64Mi limit (137)	try=2 max_tries=1	try=2 max_tries=1	invariant
k8s_timeout	execution_timeout → AirflowTaskTimeout	try=2 max_tries=1	try=2 max_tries=1	invariant
k8s_markfail	user marks running TI failed via REST API	try=1 max_tries=0	try=1 max_tries=0	invariant
k8s_sleep	running pod force-deleted (infra), repeated	try=2 max_tries=1	try=5 max_tries=4	refunded 3×, capped
k8s_mixed	infra kill on attempt 1, then app ValueError	try=2 max_tries=1	try=3 max_tries=2	user retry preserved

Two properties, both proven live:

• The 4 user/app modes are invariant. Turning the flag on changes no user-caused failure — app crash, OOM-against-own-limit, timeout, and an explicit user cancel all spend the retry exactly as today. Infra refund never masks a user failure. (RED still populates failure_details — Pillar 1 is additive — but max_tries never moves.)
• Only the 2 infra modes flip, and the cap bounds them. k8s_sleep died after 2 kills in RED (max_tries=1) and survived 5 in GREEN (refunded 1→2→3→4, then capped). k8s_mixed is the headline: the infra kill on attempt 1 is refunded, so the app code still gets its full two attempts (try 2 and 3 ran to the real ValueError) — in RED that same kill left the app only one shot.

Raw scheduler logs (verbatim)

# app OOM -> bridge classifies user, refund gate does not fire
Changing state of KubernetesResults(...dag_id='k8s_oom'... failure_details={'container_reason': 'OOMKilled', 'exit_code': 137, ...}) to failed
AIP97_LISTENER on_task_instance_failed dag=k8s_oom task=oom_task try=1 max_tries=1 failure_details=TaskFailureInfo(source='user', executor_kind='kubernetes', infra_reason='OOMKilled', infra_metadata={... 'exit_code': 137 ...})
-> retry also OOMKills -> try=2 max_tries=1 -> FAILED (retry spent by the app bug)

# infra disruption -> scheduler classifies infra, refund fires
AIP-97: infra failure (ExecutorReportedFailure) refunded attempt for <TaskInstance: k8s_sleep.sleep_task ...>; max_tries now 2 (refund 1/3), user retry budget preserved
AIP97_LISTENER on_task_instance_failed dag=k8s_sleep task=sleep_task try=1 max_tries=2 failure_details=TaskFailureInfo(source='infra', executor_kind='KubernetesExecutor', infra_reason='ExecutorReportedFailure', ...)

# DB
 k8s_oom   | failed  | try=2 | max_tries=1     <- app OOM: retry spent, not refunded
 k8s_sleep | running | try=2 | max_tries=2     <- infra kill: refunded 1->2

Unit — 13 tests: the reason→source table (Evicted/Preempting/NodeShutdown/... → infra; OOMKilled/Error → user; a node Evicted with a container OOM → infra), an end-to-end pass through collect_pod_failure_details with an OOMKilled V1Pod, and the executor→scheduler seam (get_task_failure_info clears after one read).

Out of scope

A real node-pressure Evicted (phase=Failed, reason=Evicted) needs controllable node memory, unsafe to force on a single-node kind; the external force-delete drives the same source='infra' → refund path and the precise pod-reason mapping is unit-pinned.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 22273074-3fdf-4a68-b0dc-0f3aa56c27b9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 1fanwang/aip97-k8s-bridge

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[1fanwang]

Superseded by #12 (Kubernetes classify) in the consolidated apache#66405. Closing; branch kept.