fix(scheduler): survive SQL-unavailable startup and recover hung tasks (GenRTP incident)

[svtica]

Context

On 2026-07-10 ~16:00, all servers rebooted simultaneously for Windows Updates. LiteTask started before a SQL Server was up, launched all 6 tasks in the same second, and a process hung while holding its in-process lock. The stale detector alerted once, its cleanup could not release a lock whose process was still alive, and the process was silently skipped ~6,700 times over 5 days until a manual service restart on 2026-07-15.

This PR addresses the whole failure chain, following the six reliability goals from the incident review.

Changes

1. Dependency readiness gate (new SqlReadinessChecker)

Before dispatching a SQL-dependent task, the scheduler probes SQL Server with a lightweight 5 s connection test. While unavailable, the task is deferred to the next 30 s tick (its schedule is untouched) and probes retry with exponential backoff (15 s → 10 min cap), each attempt logged. A task is SQL-dependent when it has a SQL action or its name matches Reliability/SqlGatedTasks (default GenRTP,Extracteur, case-insensitive substring match — this generalizes the site-side Extracteur_VerifierSiPossibleExecution check into a reusable gate). Safety valve: a login/database error means the engine answered, so it counts as ready — a permissions issue can never block tasks forever.

2. Per-task execution timeout with forced kill (the key fix)

TaskAction.TimeoutMinutes was persisted since day one but never enforced. Now:

• Executable/Batch children and sqlcmd run through RunProcessWithTimeout, which kills the entire process tree on expiry (the old no-timeout RunProcess path is removed).
• The PowerShell in-process timeout uses the action's configured value instead of a fixed 4 h.
• A scheduler-level backstop (action timeout + 2 min) kills the tracked child processes, releases the lock, logs an error, and lets the next schedule fire — even if the execution path itself hangs (e.g. output streams never closing).

3. Stale-lock recovery that actually kills the process

TaskRunner now tracks child processes by PID per task (task name flows via AsyncLocal, no signature changes) and exposes KillProcessesForTask. CleanupStaleTasks no longer resets the running state at 15 min — that reset is what caused the endless re-dispatch into a held lock. Instead it alerts, and past sum of action timeouts + 10 min it kills the owning process(es), force-releases the semaphore, and resets the state.

4. Escalating stale/blocked alerting

Stale alerts repeat every Reliability/StaleAlertRepeatMinutes (default 60) with an ALERT #N counter in the subject while the condition persists. A task that keeps being skipped (lock unavailable / already running) now also raises repeating alerts after 15 min of blockage, instead of thousands of silent log warnings.

5. Post-reboot startup delay + staggered launch

The service waits Reliability/StartupDelaySeconds (default 90) before the first dispatch, and tasks due in the same tick are staggered Reliability/TaskLaunchStaggerSeconds apart (default 5 s) via a pending-dispatch guard that also prevents double launches. Dispatch is offloaded to the thread pool, so the up-to-30 s lock wait no longer stalls the scheduler tick.

6. Non-interactive child execution

SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX) at service start (inherited by children) prevents Windows hard-error dialogs from blocking a child invisibly, and the PsExec -i flag is now only passed in interactive sessions. stdout/stderr were already redirected and logged; additionally ExecuteSqlTask no longer reports success when sqlcmd fails or times out.

Configuration

New optional Reliability section in settings.xml (defaults apply when absent):

<Reliability>
  <StartupDelaySeconds>90</StartupDelaySeconds>
  <TaskLaunchStaggerSeconds>5</TaskLaunchStaggerSeconds>
  <SqlReadinessEnabled>True</SqlReadinessEnabled>
  <SqlReadinessServer></SqlReadinessServer> <!-- falls back to SqlConfiguration/DefaultServer -->
  <SqlGatedTasks>GenRTP,Extracteur</SqlGatedTasks>
  <StaleAlertRepeatMinutes>60</StaleAlertRepeatMinutes>
</Reliability>

Behavior changes to review

• PowerShell tasks previously had a 4 h cap; every task now defaults to its action's TimeoutMinutes (default 60 min). Any task that legitimately runs longer needs its TimeoutMinutes raised in the task definition.
• SQL tasks now report failure (and trigger the failure notification) when sqlcmd exits non-zero or times out; previously they always reported success.
• The readiness probe uses Integrated Security under the service account (same pattern as SqlTab); the login-error-counts-as-ready rule keeps this safe even if the account has no SQL login.