AIP-97 POC: dedicated infra_retry_count counter vs refund-max_tries (open question)

[1fanwang]

Open-question POC — reads on top of #10. This is the dedicated-infra_retry_count-counter alternative to refunding max_tries. The base design is now consolidated in #10 (off the apache root); this branch still stacks on the earlier aip97-k8s-bridge head, so read it as "the counter delta vs. the design in #10," not standalone.

---

AIP-97 POC — dedicated infra counter vs refund-max_tries

This branch POCs the AIP-97 open question: refund max_tries (the shipped approach in #6/#7) versus
a dedicated infra_retry_count column that never touches the user's retry number.

The one difference

Both give the same retry behavior — an infra disruption extends eligibility so the user's code
still gets its full retries. The difference is the data model:

	Refund max_tries (#6/#7)	Dedicated counter (this)
after an infra kill	max_tries 1 → 2 (inflated)	max_tries stays 1; infra_retry_count 0 → 1
user's configured retries still readable on the TI?	no — conflated into max_tries	yes — max_tries == retries always
migration	none	one new column
retry behavior	identical	identical

is_eligible_to_retry becomes try_number <= max_tries + min(infra_retry_count, cap); max_tries
is never mutated.

Live e2e — the cap, on a real KubernetesExecutor cluster

Counter branch deployed in-cluster on kind (Postgres + api-server + dag-processor + scheduler as
pods, KubernetesExecutor, max_infra_refunds=3). Migration 0127 applies clean on real Postgres
(infra_retry_count column present). k8s_sleep (retries=1) disrupted by real pod force-deletes,
one snapshot each:

disruption	try_number	max_tries	infra_retry_count
1	2	1	1
2	3	1	2
3	4	1	3
4	5	1	3 (capped)

scheduler: infra_retry_count now 1/3, max_tries unchanged at 1 (== retries), user retry budget preserved
           ... now 2/3 ... ... now 3/3 ...   (4th disruption: at cap, no increment)

infra_retry_count climbs 1→2→3 and the cap stops it; max_tries never leaves 1. On the identical
scenario the refund branch (#6/#7) took max_tries 1→4. Retry behavior is identical (both
survive retries + cap = 4 refunded attempts); only the data model differs — the counter keeps the
user's retries legible on the row. The deterministic cap math is also unit-pinned (5 tests).

Unit tests (deterministic — the cap + pristine max_tries)

test_taskinstance_infra_counter.py, 5 cases:

• infra bumps infra_retry_count, max_tries unchanged
• cap=3 bounds it to 1,2,3 then stops; max_tries never moves
• app / user / timeout / None never count
• disabled by default
• retries=None doesn't crash

Recommendation for the AIP

Lead with the refund (minimal, no migration, ships the behavior); offer the dedicated counter as the
cleaner data model when reviewers want the user's retries to stay legible. This POC shows the
counter costs exactly one column + a migration and changes no behavior.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e4e588db-b6cf-4c4a-b401-b750e7865e05

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch aip97-poc-dedicated-counter

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[1fanwang]

Superseded by #17, the current dedicated-counter cut (open question 2, handles retries=0). Closing; branch kept.