Heal stale WAL generation in task_store reads (#889)

[olegbrok]

The bug

TaskStore keeps a long-lived thread-local sqlite handle. That handle can end up bound to a WAL generation that was checkpointed away underneath it. Once that happens, every statement on that handle raises database disk image is malformed even though the database file is intact — integrity_check passes from a separate process.

Live impact on the TOD fleet: GET /tasks 500s and get_next_task() fails until the daemon is restarted. The restart both masks the cause and eats queued scheduled work, so the failure looks transient and its evidence disappears. Onesie hit it twice in one night.

Same family as #889 and the earlier deleted-WAL split-brain incident.

The fix

Route the SELECT paths through _read / _read_one. On sqlite3.DatabaseError: drop the thread-local handle, reopen, retry exactly once.

Why reads are safe to retry: the error fires before any row is produced, so a retry cannot double-apply anything.

Write paths are deliberately untouched. A blind retry could re-apply a statement that had partially landed before the handle went bad. Hardening writes is a separate, deliberate pass — verified in this diff: zero INSERT/UPDATE/DELETE call sites were rerouted.

Failure behavior is preserved, not papered over:

• The retry is bounded to one attempt, so a genuinely corrupt database still raises rather than looping.
• Every heal is logged (task_store: read hit stale handle (...); reopening connection, retry 1/1), so self-healing doesn't hide the incident rate — this class of fix is only acceptable if it stays loud.

Provenance and verification

Found, diagnosed, and patched by angel (TOD fleet) after Onesie routed the incident to it. Angel could not run pytest or ruff on that box (neither is in its venv) and validated via a standalone harness instead — so it explicitly did not claim CI-grade verification, and did not push or open this PR itself.

Verified by me on the Mini against current main (efbb09d):

• Patch applies clean to main (angel checked it against 1bbe3d3; two merges have landed since).
• ruff check — clean.
• pytest tests/test_task_store.py — 48 passed.
• The three new recovery tests were run in isolation and read for substance rather than counted: they poison the thread-local handle with a stub that raises malformed once, prove list() and get() return correct data afterward, assert the poisoned handle was actually replaced, and — for the bound — force the reopen to also yield a broken handle so the error must propagate. That last one is the test that makes the "no infinite retry" claim real.
• Full-suite result appended below before this leaves draft.

Review notes

The judgment call worth a second opinion: sqlite3.DatabaseError is broad — it also covers OperationalError (e.g. database is locked). Such an error would trigger a drop-and-reopen that cannot clear a lock, then one retry, then propagate. Wasteful but not harmful, and the alternative (matching on the message string) is more brittle. Flagging it rather than changing it.

Closes the read-path half of #889.

🤖 Opened by Barsik