Archived sandboxes leave provision jobs grinding phantom resources; client timeouts orphan the worker

[haasonsaas]

Summary

When a client gives up on a slow provision (e.g., Orb's ORB_SANDBOXWICH_PROVISION_TIMEOUT_SECONDS expiring during a gVisor cold start) and the sandbox is archived, the worker's provision job keeps its lease and continues working on a pod/PVC the archiver deleted. Two knock-on failures:

1. The cluster-autoscaler cancels the scale-up (no Pending pod left to schedule), so the node never arrives; the worker waits out its full kubectl timeout on a phantom pod.
2. The single-replica worker then serially grinds each stale job to its timeout (here ~15–25 min each with SANDBOXWICH_KUBECTL_COMMAND_TIMEOUT_SECS=900), starving new provision jobs that could have succeeded on the warm pool.

Evidence (production, evalops-prod, 2026-07-18 06:39–07:30Z)

Four Orb spawn attempts each created a job; Orb timed out at 180s and archived each sandbox mid-provision. Observed: pods deleted while their jobs stayed leased, CA scale-up triggers (06:44, ~07:10) cancelled with the node MIG returning to target 0, and later jobs queued behind the stale ones.

Expected

• Archiving/deleting a sandbox cancels (or fences) its pending/leased provision jobs, and the worker skips to the next job instead of timing out on deleted resources.
• Optionally: provision jobs carry a deadline so a client-side timeout can be honored server-side.

Related: #181 (the same cold-start path also dead-ends retries with the durable-identity 409).