fix(serving): take the api_usage write off the request path

[brownjuly2003-code]

What

Two commits:

1. 737795d — docs: state the real Spaces quota. The docs claimed the free tier keeps three cpu-basic Spaces awake. It keeps four running under this account today, and POST .../restart on a paused one is refused with an explicit quota error. A second account is not an escape hatch: creating another free Docker Space returns 402. Closes S5 through the DoD's "or quota is stated" branch, with the measured constraint rather than the assumed one.

2. ab7257d — fix: take the api_usage write off the request path. This is finding N1, and it turned out to be a code defect rather than runner noise.

N1 — the Load Test was not bimodal because of the runner

Every authenticated request wrote its own api_usage row before the response was produced. The embedded store serializes writers and commits per row, so each request queued behind one fsync and the API was capped at 1 / commit_latency rps.

The load client is closed-loop (15 users, between(0.1, 0.5) think time), so it has two equilibria: an unsaturated one bounded by think time (~46 rps), and a saturated one pinned at rps = 1/s — independent of user count. The runner's disk only decided which side of s ≈ 20 ms the commit landed on.

Two things falsify the "slow runner" reading:

• rps varies 1.5× across runs while p99 varies 10×. A 1.5× slower machine gives a 1.5× tail, not a 10× one. That nonlinearity is a queue.
• The red branch is an attractor, so its runs agree: 29.4 · 29.1 · 28.9 rps (spread 1.7%). The nine green runs spread across 37.0–46.2 (25%).

Reproduced on the shipped code

Serving the real API with the CI load profile and sleep(s) injected into record_api_usage:

s	in-path	off-path
0 ms	43.9 rps · p99 100	—
34 ms	31.4 rps · p50 160	37.9 rps · p99 340
60 ms	8.3 rps · p99 21000 · FAIL	37.2 rps · p99 320 · PASS

CI's red branch is 29.1 rps / p50 161 ms. Off-path throughput is flat in s; in-path throughput is 1/s.

The fix

The request enqueues a UsageRow and returns; one background thread drains the queue.

Batching is part of the fix, not a tuning knob — a per-row background writer still commits 1/s rows per second, below the request rate, so the ceiling would move from request latency into a queue that silently overflows. record_api_usage_batch puts one commit under N rows.

What it trades

Durability moves from "committed before the response" to "committed shortly after". A crash loses at most the queued rows. api_usage backs one admin read (GET /v1/admin/usage) — not billing, not rate limiting — and rows were already droppable when the store exhausted its retries. Reads that must see their own writes call flush_usage; the lifespan closes the writer on shutdown. A full queue sheds rows into agentflow_usage_rows_dropped_total rather than stalling the request it was counting.

Tests

tests/unit/test_usage_write_off_request_path.py pins the guarantee, not the mechanism: a slow write must not delay the response (verified to fail when the write is put back on the path), a failing write still serves the request and counts the drop, a failed batch publishes no audit event, batching coalesces, a full queue sheds, and flush gives read-your-writes.

test_auth_usage_write_failure.py is removed. It drove its failure through AuthManager.record_usage, which the request path no longer calls, so it guarded a call rather than a promise; that promise is now test_request_succeeds_when_the_usage_write_raises.

Verification

• pytest tests/unit — 1678 passed. The one failure, test_version, reproduces on a clean tree (stale agentflow-client 1.6.0 in the local venv) and is not touched by this diff.
• ruff check src/ tests/ scripts/ sdk/ — clean; ruff format --check — 329 files formatted.
• The real end-to-end check is this PR's own Load Test run.

Full measurement: docs/perf/usage-write-bifurcation-2026-07-09.md.

🤖 Generated with Claude Code

[github-actions]

DORA Metrics

• Window: last 30 days
• Branch: main
• Deployment frequency: 95 total / 22.17 per week
• Lead time for changes: avg 0.43h / median 0.0h
• Change failure rate: 61.05% (58/95)
• MTTR: n/a across 0 incident(s)

MTTR note: No failed mainline CI runs in the selected window.