fix(archiver): replace stale jobs so retry can escape a wedged download

[dylanjeffers]

Closes #81.

Problem

Job ids are deterministic — base64url("{userId}-{trackId}") — and getOrCreateStemsArchiveJob only replaced a job whose state was failed:

const state = await existingJob.getState()
if (state !== 'failed') {
  return getJobStatus(jobId, existingJob)   // same stuck job, every time
}

So any job parked in a non-terminal state is handed back to every subsequent retry from that user for that track.

This is what made the 2026-07-28 outage (#77) unrecoverable from the client. When archiver-redis was recreated the worker lost its locks, kept all five concurrency slots, and jobs sat in waiting — never failed. The "Try again" link in DownloadTrackArchiveModal is the only recovery affordance the UI offers, and it was a no-op: POST returned the same dead job id, the client re-polled it, and timed out again 15 minutes later. 24 requests across 15 tracks were stuck this way for ~7 hours.

Cancel-then-retry from the client doesn't work around it either — cancelStemsArchiveJob removes a job from the queue only when it's completed; otherwise it aborts via an in-memory AbortController, and a job stranded in waiting has none, so it falls through to 'Stems archive job not found' and removes nothing.

Change

A job that has been non-terminal past staleJobSeconds is now removed and recreated. Behavior by state:

state	before	after
completed	returned	returned (unchanged — download the ready archive, don't rebuild)
failed	replaced	replaced (unchanged)
non-terminal, fresh	returned	returned (unchanged — this is what dedupes double-clicks / multiple tabs)
non-terminal, stale	returned	replaced

Staleness is measured from processedOn ?? timestamp, so a job that never started — the case that actually broke — is covered, not just one that stalled mid-run.

New config staleJobSeconds (ARCHIVER_STALE_JOB_SECONDS, default 900s). Chosen to match STEMS_ARCHIVE_POLL_TIMEOUT_MS in the client's useDownloadTrackStems: past that point the client has already given up on the job, so its output would never be collected anyway. The coupling is called out in a comment on both sides.

Notes for review

• Same job id is reused on replacement. This matches what the existing failed path already did. processJob reuses /tmp/audius-archiver/{jobId} if it exists and stem filenames are deterministic, so a replacement overwrites rather than colliding. Minting a fresh id would leave the old temp dir behind until cleanupOrphanedFiles swept it, and would break dedupe.
• Not included: letting cancelStemsArchiveJob remove queued/active jobs. It's a real gap (noted in archiver: retry can never escape a stuck job — getOrCreate only replaces failed jobs #81) but a separable change; this PR stays on the retry path.
• Doesn't fix the underlying wedge — that's archiver: worker never releases job slots after Redis lock loss, wedging the queue #77. This makes the wedge recoverable by the user instead of terminal.

Testing

New src/jobs/createStemsArchive.test.ts, 11 cases: isStaleJob boundaries (processedOn preferred over timestamp, fallback for never-started jobs, missing timestamps treated as fresh) and each getOrCreateStemsArchiveJob branch, including the exact wedged-in-waiting case from the outage.

Test Files  3 passed (3)
     Tests  31 passed (31)

tsc --noEmit clean. ESLint could not be run from a git worktree — eslint-plugin-turbo throws Cannot convert undefined or null to object outside a turbo workspace root, identically on unmodified main, so it's environmental and left to CI.

🤖 Generated with Claude Code