trust-factor W re-measurement: contention-safe sweep harness

[cdeust]

Summary

Continuation of #410 (already merged — the ingest_memory() capture_origin
fix). This PR carries the harness hardening required to re-measure the
trust-factor W grid on a machine that was, for most of the night, shared
with other agents running full test suites — plus the re-measured grid
itself once it completes.

What's in this PR so far

• benchmarks/lib/machine_load_snapshot.py (new, split out of
  write_manifest.py to satisfy the local craftsmanship gate's 40-line
  method cap) — load average, concurrent pytest processes, concurrent
  Docker containers.
• benchmarks/lib/disk_space_snapshot.py (new) — free/total bytes on the
  repo-root filesystem and Docker's storage root. A neighboring session
  sharing this machine reported leaked test databases filling the host
  disk to 100% on 2026-08-09, taking PostgreSQL down mid-measurement — the
  same invisible-failure shape as CPU contention, a different resource.
• benchmarks/reproduce.sh / write_manifest.py — both snapshots are
  taken at cell START (before start_db) and cell END, recorded in every
  MANIFEST.json alongside git_sha.
• 33 tests across the touched files, all green. ruff check/format
  clean. python scripts/check_craftsmanship.py clean on every touched
  file.

What's still coming (same PR, do not merge yet)

• The re-measured 5-cell W grid (1.0/0.8/0.7/0.6/0.5), including the
  control cell W=1.0 reproducing the pre-recall fuses relevance and heat only — no trust or provenance dimension in WRRF, so a poisoned memory outranks a good one #368 published numbers.
• Point-by-point effect on rank (per-cell MRR/R@10, overall + per-category).
• docs/provenance/trust-factor-calibration.md updated with a new dated
  section stating explicitly what invalidated the old gated arm
  (capture_origin never set → uniform demotion) and what this
  re-measurement establishes.

Background (why a harness PR before the numbers)

Two sweep attempts were discarded outright before this branch existed —
not because a cell crashed, but because contention (CPU load above the
core count, confirmed via ps aux/uptime) can produce a cell that
finishes cleanly and still returns degraded numbers, indistinguishable
from a clean run without instrumentation. Per this project's own rule, a
measurement from a harness carrying a known defect is invalid and gets
redone, not patched after the fact. The load/disk snapshots in this PR are
that instrumentation — every future cell now self-documents whether it ran
clean.

Test plan

• uv run --no-sync pytest -q on the touched files — 33 passed
• python scripts/check_craftsmanship.py <touched files> — clean
• ruff check / ruff format --check — clean
• Full grid re-measured on a confirmed-clean machine
• docs/provenance/trust-factor-calibration.md updated
• CI green

Co-Authored-By: Claude noreply@anthropic.com

[cdeust]

ZETETIC-REVIEW: APPROVE

Head reviewed: fb28eb54 (current HEAD, trust-factor-sweep-remeasure) — a formatting-only commit landed after my initial pass at 2cbe10e3 (ruff format fix for a line the earlier commit skipped, per the PR's own commit message; git diff 2cbe10e3..fb28eb54 confirms only sweep_progress.py, 1 insertion/3 deletions, no logic change). All findings below are re-verified against fb28eb54.

Move 0 — Ledger reconciliation / seen-defect check

No unmapped branches, no seen-defect rationalization phrases ("pre-existing," "unrelated," "out of scope") anywhere in the diff or PR body. Pass. (Note: the PR description itself is stale — written mid-flight with unchecked "still coming" boxes — see Non-blocking below; this is a hygiene gap, not a rationalized defect.)

Stakes (Move 7)

Medium — docs/provenance/trust-factor-calibration.md is a published claim about the shipped system (the calibration source-comment for UNTRUSTED_ORIGIN_FACTOR in mcp_server/core/retrieval_dispatch.py cites it), but no production/runtime code is touched by this diff — retrieval_dispatch.py itself is unchanged, W stays 0.7. benchmarks/lib/* is internal tooling, not user-facing.

The five verification points

1. Table numbers vs. committed artifacts — verified exactly, all five cells:

W	Doc claims (MRR / R@10)	Artifact (longmemeval-s.json/locomo.json/beam-100K.json)	git_sha
1.0	LME .8500/1.0000, LoCoMo .8209/.9594, BEAM .6579	.85/1.0, .82087/.95939, .65792	f87bf6e3
0.8	LME .8500/1.0000, LoCoMo .8043/.9391	.85/1.0, .80428/.93909	f87bf6e3
0.7	LME .8500/1.0000, LoCoMo .8044/.9340	.85/1.0, .80441/.93401	f87bf6e3
0.6	LME .8500/1.0000, LoCoMo .8044/.9340	.85/1.0, .80441/.93401	f87bf6e3
0.5	LME .8500/1.0000, LoCoMo .7892/.9188	.85/1.0, .78918/.91878	f87bf6e3

All five MANIFEST.json report the same git_sha (f87bf6e3), same longmemeval_dataset_sha256, same embedding_model_revision, reranker_active: true — one shared provenance, as claimed. Sample sizes confirmed directly in the JSON: n_questions=10 (LME), n_conversations=1/n_questions=197 (LoCoMo) — matches reproduce.sh's --quick per-suite limits (--limit 10/--limit 1/--limit 2, reproduce.sh:421) exactly. The PROGRESS.json/repro_dir.txt mapping in active/cell_W* matches the doc's Artifacts column 1:1. The load-average claim ("6.2–13.1") is exact: computed min/max across all ten start/end snapshots = 6.16/13.12.

2. Cost arithmetic — recomputed independently: 0.8208665699782451 − 0.8044053662073968 = 0.016461... ≈ 0.0165 MRR ✓; (0.9593908629441624 − 0.934010152284264) × 100 = 2.5381... ≈ 2.54 points R@10 ✓. Both match the doc exactly.

3. Both reserves stated, not buried — confirmed. Quick-mode/floor-gate reserve is item 1 under a dedicated "### What this does NOT establish" heading immediately after the results table, cites reproduce.sh:94-95 verbatim (--limit/--quick runs "skip the gate... partial runs are not comparable to n=500/n=1986"), verified against the script directly — accurate. Adversarial-defense-not-linked-to-relevance reserve is item 2, same section, explicit and unambiguous ("No measurement in this document links the adversarial-arm defense rate to a benefit on the real benchmarks"). Neither reserve is a footnote; both are structurally prominent.

4. Self-contradiction check (the #414 failure mode) — the retained original diagnosis is honest, not smuggled-in validation. The original "### Decision" section (pre-fix) still reads "no floor was relaxed... W = 0.7," but it is immediately followed by "### What this measurement does and does not establish," which correctly identifies that the gated arm's floors could not discriminate W at all because every benchmark memory landed on capture_origin='unknown' and a uniform rescale cannot change WRRF order — stated plainly, not softened. The later "## Re-measurement" section explicitly supersedes it: "The paragraph above diagnosed a defect, not a residual unknown." I found no sentence anywhere in the document that re-presents the old non-discriminating gated-arm PASS as validation of W=0.7 after that point — the final "### Decision (owner call...)" section grounds the choice in the cost being accepted, not in the old floors. This clears the #414 bar.

5. Reproducibility from the branch alone — benchmarks/lib/sweep_progress.py (new, resumable per-cell checkpoint/retry), disk_space_snapshot.py, machine_load_snapshot.py (new) and the trust_factor_sweep.sh rewrite (one-cell-per-invocation, resume from PROGRESS.json) are all committed alongside the five repro/ result directories and the active/ sweep state. Ran the targeted suite directly from a fresh venv on this head: tests_py/benchmarks/ → 83 passed, 2 skipped (skips are pre-existing Postgres-gated tests, no Postgres in this sandbox). ruff check . / ruff format --check . → clean. scripts/check_craftsmanship.py --base origin/main → OK. scripts/check_marketplace_pins.py → current (one pre-existing, unrelated MCP-registry propagation NOTICE).

Contract drift note (non-blocking)

mcp_server/core/retrieval_dispatch.py:29-38 (untouched by this diff) still cites only the original, now-known-non-discriminating gated-arm justification ("4/4 PASS... 0.7 defends 4/4 while all four gated floors hold") as its source: comment for UNTRUSTED_ORIGIN_FACTOR = 0.7, without pointing at the newer "Re-measurement"/owner-call section that is now the operative justification (the old one is real but incomplete — it's the pre-fix data). Since the file is outside this diff's changed-file set and the constant's value is unchanged, this is not this PR's defect to fix, but it is the same class of drift Move 3 exists to catch: recommend a fast follow-up updating that comment to cite §Re-measurement alongside §Results.

Move 4 — test adequacy

disk_space_snapshot.py and machine_load_snapshot.py have direct unit tests (test_disk_space_snapshot.py, test_machine_load_snapshot.py), and write_manifest.py's extraction (_environment_fields, _start_snapshot_fields) is covered by the rewritten test_write_manifest_machine_load.py. benchmarks/lib/sweep_progress.py (new: read_progress, completed_w_values, next_pending_w, record_cell_result — real resume/replace-cell branching logic) has no dedicated unit test; it's only exercised indirectly through the shell script's inline python -c calls during an actual sweep run. Non-blocking (dev-tooling, not production path, craftsmanship gate does not require it here) but worth a follow-up test for the resume/replace-on-rerun branch.

Non-blocking

• PR description is stale: written mid-flight, "What's still coming" checkboxes still unchecked though the diff now contains the completed re-measurement and doc update. Recommend updating the description before merge so it matches the shipped diff.
• retrieval_dispatch.py:29-38 source comment (see Contract drift note above).
• benchmarks/lib/sweep_progress.py lacks a unit test for its resume logic (see Move 4).

Verdict

APPROVE. All five committed table values verified exactly against their JSON artifacts; the cost arithmetic checks out to the fourth decimal; both reserves are stated prominently, not buried; the retained original diagnosis is honestly superseded rather than left to imply a false validation; the tooling is reproducible from the branch alone and its targeted tests, ruff, and both repo gates are clean on the current head. W=0.7 remains the owner's call, correctly published with its known cost and its known gaps.