fix: workers use mutex table for SP lock

[SgtPooki]

Summary

• Replace the per-session advisory lock with a DB-backed per‑SP mutex to prevent concurrent deal/retrieval jobs for the same SP across all workers.
• Record the worker hostname and job ID with the mutex for observability and stale-lock recovery.
• Add runbook guidance for capacity and backlog limits.

Problem

Staging showed spikes (e.g., 72 deals in flight) despite a “per‑SP” lock. The advisory lock was session-scoped, so with pooled connections and per-worker concurrency (10 deals + 10 retrievals), multiple jobs could run against the same SP on a single worker. With 5 workers, this allowed many concurrent deal/retrieval jobs for the same SP.

Fix

• Introduced a job_mutex table with a unique constraint on sp_address.
• Replaced advisory lock acquisition/release with mutex insert/delete using job ID and hostname.
• Added stale-lock replacement via JOB_LOCK_STALE_SECONDS.

Notes

• Lock remains global per SP (deal + retrieval share the same mutex).
• Concurrency is now limited by SP count rather than per-worker sessions.
• See docs/runbooks/jobs.md for capacity formulas and example limits.

[Copilot]

Pull request overview

This PR replaces the previous session-scoped per‑SP advisory lock with a DB-backed per‑SP mutex to prevent concurrent deal/retrieval execution for the same provider across all pg-boss workers, and updates operational docs accordingly.

Changes:

• Add job_mutex table + repository methods to acquire/release a per‑SP mutex with stale-lock takeover.
• Update pg-boss workers to use the new mutex (recording job ID + worker identity) and requeue on contention.
• Document capacity/backlog sizing guidance and introduce JOB_LOCK_STALE_SECONDS configuration/docs.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/runbooks/jobs.md	Adds operational guidance and capacity/backlog formulas; notes global per‑SP mutex behavior.
docs/infra.md	Updates infra notes to reflect DB-backed per‑SP mutex with multi-replica workers.
docs/environment-variables.md	Documents new JOB_LOCK_STALE_SECONDS env var.
apps/backend/src/jobs/repositories/job-schedule.repository.ts	Implements SQL-based acquire/release for per‑SP mutex via job_mutex.
apps/backend/src/jobs/jobs.service.ts	Switches worker handlers from advisory locks to DB mutex; adds worker identity + stale-lock config usage.
apps/backend/src/jobs/jobs.service.spec.ts	Updates unit tests to mock/expect mutex acquisition calls.
apps/backend/src/database/migrations/1760000000003-AddJobMutex.ts	Adds the job_mutex table and indexes.
apps/backend/src/config/app.config.ts	Adds config validation/loading for JOB_LOCK_STALE_SECONDS.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.