Worker pod-replacement retry dead-ends on API durable-identity 409 after transient pod-ready timeouts

[haasonsaas]

Summary

When the kubernetes provider's pod-ready wait times out (classified kubernetes_provider_transient), the worker's retry deletes and recreates the sandbox pod. The replacement pod has the same name but a new UID. The next provisioning-stage report then fails permanently with:

409 Conflict: {"ok":false,"code":"provisioning_resource_identity_conflict","message":"durable resource identity cannot change within a stage"}

So a transient, recoverable timeout becomes a permanently failed job — worst case, every retry of a cold-start provision fails this way.

Evidence (production, evalops-prod, 2026-07-18 ~06:40–07:20Z)

Worker: sandboxwich-worker@sha256:95633cc2… (rev 6f64900). API same rev.

1. Orb created sandbox 019f73f3-e684-… (tenant orb, template orb-sandboxwich-runtime@sha256:fe4814d3…).
2. Pod sat Pending through the gVisor pool's scale-from-zero (~11 min to node ready).
3. Worker reported kubernetes_provider_transient: sandbox pod did not become ready at ~06:54Z.
4. Pod was replaced; subsequent stage report → provisioning_resource_identity_conflict → job permanently failed.

The guard lives in crates/sandboxwich-api/src/handlers/leases.rs (~line 663): same (sandbox_id, stage, kind, namespace, name) with a different resource_uid/generation is rejected. A pod replacement after a transient timeout is a normal k8s recovery move, not identity drift.

Expected

Transient pod-ready timeouts recover cleanly: either the provider waits on the existing pod (no delete/recreate), or the API accepts a same-name replacement UID as stage progress (e.g., allow UID update when the previous resource is gone, or compare by name only for Pods).

Repro

1. Scale the gVisor pool to zero.
2. Create a sandbox (cold start forces a pod-ready timeout).
3. Watch the job: transient failure → pod replacement → permanent 409.