fix: jobs can remain permanently active if final status persistence fails after three short retries

[sam-saffron-jarvis]

What changed

• Reworked finishRunWithRetry in cmd/serve_jobs_v2.go to retry final run persistence for up to 30 seconds instead of only 3 short attempts.
• Added bounded exponential backoff between retries, starting at 50ms and capping at 1s.
• Stop retrying early if the manager is shutting down, and log that restart recovery may still be needed.
• Strengthened the existing transient SQLite lock test so it now holds the lock long enough to exceed the old 3-attempt / 150ms retry window.

Why this is high-value

A run only leaves queued / claimed / running through final status persistence. Before this change, a brief burst of SQLite write failures at completion time could leave a run non-terminal forever until process restart. That could permanently consume concurrency slots and block future job executions, especially for single-run or forbid jobs.

This change keeps terminalization durable across transient DB write failures, preventing a single short-lived persistence error from wedging the job pipeline.

Validation

• gofmt -w cmd/serve_jobs_v2.go cmd/serve_jobs_v2_test.go
• go build ./...
• go test ./...
• Updated TestJobsV2ExecuteRunRetriesFinishRunAfterTransientDBLock passes with a lock duration that exceeds the old retry budget.

[SamSaffron]

Implemented in local copy: jobs v2 final run persistence now retries for a bounded 30s window with exponential backoff and exits early during shutdown. Verified with go build ./... and targeted tests.