Worker heartbeat is interleaved with the claim loop; lease execution is one-at-a-time

[haasonsaas]

Problem

Two related worker-loop limitations in crates/sandboxwich-worker/src/main.rs's work_loop:

1. Heartbeat is not an independent task. The heartbeat call runs inline at the top of the same loop that claims and executes leases. If heartbeat_worker fails after retries, the current iteration is abandoned (continue) before a claim is even attempted — so a heartbeat hiccup delays lease claiming, and conversely a long-running lease delays the next heartbeat, since both share the same loop iteration cadence.
2. Lease execution is strictly one-at-a-time. Each iteration claims a single lease, then tokio::select!s on handle_lease(...) (racing only the drain watchdog) before looping back to claim again. There is no concurrent execution of multiple leased jobs by one worker process.

Evidence

• crates/sandboxwich-worker/src/main.rs:1462-1470 — heartbeat call inline at the top of work_loop's loop { ... }; on failure: eprintln!("error: heartbeat failed after {API_RETRY_ATTEMPTS} attempts, skipping this iteration: {error:#}") followed by continue, skipping the claim for that iteration.
• crates/sandboxwich-worker/src/main.rs:1564-1567 — a single handle_future = handle_lease(client, api, lease, provider.clone()) is raced only against drain_watchdog(...) via tokio::select!; the loop does not proceed to claim another lease until this one resolves (or the process is draining).

Suggested approach

• Move the heartbeat onto its own tokio::spawned periodic task (independent interval, independent retry/backoff), decoupled from the claim/execute loop, so a slow lease doesn't starve heartbeats and a heartbeat retry doesn't starve claims.
• Allow the claim loop to hold multiple in-flight handle_lease futures (e.g. a FuturesUnordered or a bounded semaphore-gated spawn per lease) instead of awaiting one lease to completion before claiming the next, so one worker process can execute several leased jobs concurrently up to a configurable concurrency limit.

Relationship to ROADMAP

Related to the Lifecycle recovery promotion gate ("Provision, command, stop, snapshot, fork, cancellation, lease loss, worker restart ... have deterministic terminal states") — an independent heartbeat task reduces the chance of a worker being marked stale purely because it's busy executing a single long lease.