Skip to content

Take mark_pod_claimed off the warm-claim hot path with a real reservation protocol #893

Description

@stubbi

Context

The warm-claim stage instrumentation (claim stage timing) puts the optimistic-lock pod-claim label patch at a measured 14.8 ms P50 / 20.4 ms P90 (n=1788, prod, 24h window ending 2026-07-10T19:40Z) of serialized apiserver write on every claim, second only to activate_rpc. The prepare-time-restore plan (docs/superpowers/plans/2026-07-10-prepare-time-restore.md) explicitly scopes this out: taking it off the hot path needs a reservation protocol, not a reordering.

Once prepare-time restore lands (activate becomes policy + handshake + serve), the reconcile total is roughly select_pod + mark_pod_claimed + activate + status_write_ready; the label patch then becomes about a quarter of the remaining controller path. Against the hosted TTI target (beat 136 ms and then 95.9 ms end to end) this 15 ms matters.

Measured stage budget for a successful claim (prod, same window, n=33 Ready claims)

stage P50 ms P90 ms
select_pod 0.2 0.3
resolve_secrets 0.0 0.0
mark_pod_claimed 14.8 20.4
dial_tls 0.8 1.8
activate_rpc 98.2 116.3
token_secret_wait 0.0 0.0
status_write_ready 8.7 12.0
total 137.1 154.1

Sketch

The same overlap pattern the token Secret write already uses:

  1. Reserve the selected dormant pod in a leader-scoped IN-MEMORY set. Leader election already serializes claim reconciles, so this set, not the label patch, is what actually prevents two claims taking one VM within a single leader's tenure.
  2. Fire the resourceVersion-guarded label patch CONCURRENTLY with the activate RPC.
  3. JOIN the patch before the Ready status write (the durability contract is unchanged: a claim is never Ready without its claim label durable). A patch conflict at join time means an external actor touched the pod mid-activate; fail closed exactly as an activate failure does today (release, pend, retry on a different pod).

Hazards that make this a protocol, not a reordering

  • Controller crash between activate and the label patch landing. The husk is activated but the pod carries no claim label, so pod selection sees it as dormant. The stub state machine refuses a second activate (not dormant), which fails closed, but the pod is leaked until a reaper learns to reap activated-yet-unclaimed pods. That reaper rule must ship WITH the overlap.
  • Leader handover. The in-memory set is empty on a new leader. Anything reserved-but-unlabeled at handover looks dormant. Same reaper rule covers it, but the window must be reasoned through with lease timings.
  • The rejected-claim path must release both the in-memory reservation and any landed label, in that order, or retries starve.

Acceptance

  • mark_pod_claimed no longer appears as a serialized stage on the happy path (a joined stage like token_secret_wait, normally ~0, replaces it).
  • envtest coverage for: conflict at join fails closed; crash-shaped orphan (activated, unlabeled) is reaped; a rejected claim releases the reservation.
  • The claim mutual-exclusion invariant statement in the reconciler comments is updated to name the in-memory set plus durable-label-before-Ready pair.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions