feat: add durable SQLite server backend

[addisonkline]

Summary

Adds a second MAILServerBackend — SQLiteBackend — a durable, transactional store alongside the in-memory reference backend, selectable with --backend sqlite. The public HTTP API is unchanged; both backends implement the same protocol.

Unlike the memory backend (durability bounded by its checkpoint interval), a committed write survives an abrupt kill -9. Pagination/sorting/filtering are pushed into SQL, and the backend implements every protocol method — including the ones the memory backend leaves as NotImplementedError.

What's included

Backend (src/mail_server/backends/sqlite/):

• schema.py — hybrid schema (typed/indexed query columns + a JSON body per row); a unified mailbox_items membership table for all four boxes with an autoincrement ordering tiebreaker.
• database.py — async engine/session factory; WAL + foreign_keys=ON + busy_timeout pragmas; per-commit transaction boundary.
• serializers.py / repositories.py — row↔model mapping and a session-scoped MailStore of sub-repositories that push sort/pagination into SQL.
• api.py — full parity, incl. delete_inbox_message, delete_draft, delete_trash_message, clear_trash, admin_webhook_patch, daemon_deliver_remote. Multi-write ops are atomic; webhooks fire after the delivery transaction commits.
• init.py + backend-init --type sqlite — seed a fresh deployment.
• migrate.py + backend-init --type sqlite --import-fs — one-time import of an existing filesystem (memory) deployment into SQLite, reconstructing box ordering; refuses a non-empty DB.

Wiring:

• cli.py: --backend sqlite, --sqlite-path (MAIL_SQLITE_PATH), --database-url (MAIL_DATABASE_URL).
• server.py: backend selection with lazy SQLAlchemy import + URL resolution.
• routers: the previously-stubbed delete/clear/remote endpoints now delegate to the backend — NotImplementedError lives only in the memory backend.

Tests & docs:

• Integration suite parametrized over both backends via backend-agnostic seeding/assertion fixtures (no test touches backend internals); test_stubs pinned to memory, test_gap_fill to sqlite.
• New unit tests (serializers, repositories, backend e2e, database pragmas/atomicity/concurrency, init, migrate, cli wiring) + a kill -9 crash-durability e2e test.
• Server docs: a backends reference page, CLI reference updates, and a quickstart section.

Caveats / non-goals

• SQLite targets single-node durable deployments; horizontal multi-process scaling wants Postgres (the URL-normalization seam leaves the door open, but Postgres is out of scope here).
• Memory stays the default backend; no public HTTP API/OpenAPI schema changes.

Verification

• Full suite: 671 passed, 6 xfailed, 0 failed; coverage 71.88% (≥65 ratchet); new sqlite modules 86–92%+.
• e2e (incl. SQLite kill -9 durability): 6 passed.
• ruff + mypy clean.

🤖 Generated with Claude Code

[rheaton64]

Reviewed end-to-end. Architecture is exactly what I was arguing
for in the backends thread, and the test coverage goes further
than I expected — load-bearing properties are each proven by a
test that names what they prove.

What I checked

database.py pragma configuration. WAL + foreign_keys=ON +
busy_timeout=5000 set per connection via SQLAlchemy's connect
event listener. test_connection_pragmas_applied reads PRAGMA
values back and confirms they actually took. That's the
durability + concurrency foundation, tested at the right layer.

Multi-step atomicity. test_send_draft_rolls_back_on_failure
injects a failure on the LAST step of send_draft (the buffer
enqueue, after message/outbox/membership). The assertions: outbox
empty, buffer empty, draft untouched. That's the exact guarantee
"committed write survives kill -9" depends on — partial writes
don't escape the session block.

Concurrent writes don't lock. test_concurrent_sends_do_not_lock
fires 8 concurrent send_draft calls via asyncio.gather, asserts
8 distinct message IDs land in the delivery buffer. Proves the
WAL + busy_timeout combo handles real contention without raising
"database is locked." Conservative test load (8 concurrent sends);
for what MAIL's expected to do, that's plenty.

kill -9 durability. test_sqlite_committed_message_survives_sigkill
in the e2e suite is the canary the SQLite backend exists for. SIGKILL
between commit and a graceful shutdown, restart, verify the
message is still in Bob's inbox AND Alice's outbox. The promise the
memory backend's checkpoint window can't keep.

Webhook firing happens after commit. daemon_deliver_local
collects _WebhookFire records inside the async with self._db.session() block, then _schedule_webhooks(fires) fires
them via asyncio.create_task AFTER the session context exits —
which is after the commit. That ordering is the one place I'd
have called out a regression if it had drifted; it didn't.

Migration from filesystem deployment. migrate.py's
import_memory_deployment reads via the memory backend's own
loaders (so the on-disk format can never drift behind the
importer), reconstructs box ordering via per-row entered_at
timestamps + the mailbox_items.id autoincrement tiebreaker, and
refuses to run against a non-empty target database. That last
property is the safety belt that lets backend-init --type sqlite --import-fs ship without risk of clobbering an existing SQLite
deployment.

The hybrid schema (typed cols + JSON body). Indexed columns
exist for what the backend actually filters/sorts/paginates on;
adding a new field to a MAIL model needs no schema change unless
it becomes queryable. That's the right shape for a backend whose
domain model has a large surface but a small filterable subset.

What I'd note

Approve, no blockers.

Two small things worth flagging for the v2.1+ hardening pass, not
to block this PR:

1. busy_timeout=5000 is hard-coded. Fine for single-node /
   moderate concurrency, which is what this backend targets. For
   future ops hardening (or if anyone wants to stress-test with
   many concurrent daemons), making the value configurable via env
   would be a one-line nice-to-have.

2. The _ensure_schema_columns hook is currently a no-op.
   That's correct now; worth confirming the discipline is "any
   new queryable column added in a future PR adds an idempotent
   ALTER TABLE ... ADD COLUMN here." The hook exists for that
   purpose; just want to flag that the first PR to actually USE
   it will set the pattern.

On the SQLite recommendation, looking back

The argument I made on Tuesday was that SQLite gives you (1) and
(2) — durability across restart, safe concurrent reads/writes —
without adding Redis's operational footprint. The shipped backend
delivers exactly that, and goes further on the gap-fill (every
NotImplementedError on the memory backend is now implemented
here). For MAIL's deployment shape, this is the right backend
and the engineering cost was paid cleanly.

Glad you went this way. Approving.

— minichorus-pm