fix(dm,qa): player-facing prose is FICTION-ONLY — close the OOC-narration-leak class (#1 craft lever) - #972
Merged
Merged
Conversation
…tion-leak class (#1 craft lever) A 2026-06-17 adversarial craft audit (10 reads × 2 authored spines, both scored story-craft 4.8 by a single LLM scorer) returned the verdict that the 4.8 is INFLATED (real range 4.4-4.7): the prose is genuinely BG3-tier, but the scorer blends away a recurring PRODUCT DEFECT that ALL automated gates missed — OOC DM bookkeeping leaking into the player-facing narration as standalone beats. In gs-ember-deep, 5 verbatim first-person authoring lines shipped to the player: "Now let me seat Kield Vant as the player character", "Continuity check — … let me correct that", "Let me set the order of it", "Here's how round one actually went:", "let me set their advancement through the engine". This is THE gap between the prose the spines write (4.5-4.7) and the felt experience. Two-part fix (the viewer sanitizeNarration defense-in-depth is the sibling lane → flagged separately): 1. SOURCE — skills/dungeon-master/SKILL.md FICTION-ONLY rule broadened from 3 leak families to 5: + first-person authoring/bookkeeping preambles (the #1 defect — and the note that they ride on the BACK of good prose as a trailing clause, e.g. "…the room tips into violence. Let me set the order of it."); + raw system vocabulary in prose ("natural 1"/"d20"/"DC"/HP totals, and the truncated "your nat…" stutter); + a SESSION-CLOSE clause (no author's-note/craft-jargon/raw-mechanics coda after end_session — that goes in the player-invisible summary arg). The 5 verbatim lines are now named negative examples. 2. BACKSTOP — qa/assert_behavioral.py gains narration_no_ooc_leak: the deterministic gate the LLM scorer can't be trusted for. Proportionate — 0 leaks PASS, 1-2 incidental WARN, >=3 in a substantial run RED (caps the lenses, de-inflating a leak-riddled run). Patterns are HIGH-CONFIDENCE OOC-only and were adversarially FP-hardened (machine-checked sweep): the bare `through the engine` (matched Gond/artificer/Steel-Watch machinery) is now verb/noun-anchored to the game-engine sense; `as the pc` (collided with in-world "PC" initialisms) is full-word only; `your nat…` (fictional stammering) and `spine hook` (a literal flensing tool) dropped from the gate (still SKILL-banned at the source). Verified on real data: gs-ember-deep → 5 leaks → RED (the gate now catches the inflation); gs-ledger-deep → 1 → WARN (the cleaner spine). Note: this changes gate behavior repo-wide — leak-riddled runs now go behavioral=RED, which is the intended de-inflation. 8 new tests (5 verbatim leaks match; the machine-checked machinery/PC/stammer near-misses stay clean; WARN vs RED threshold; MIN_BEATS guard). 46/46 in test_assert_behavioral.py; fast_gate 222. Follow-ups (separate): the Bless multi-target rider engine bug (cast_spell applies effect_riders to only one holder) — my lane, next PR; the viewer sanitizeNarration arm — sibling lane, flagged.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds an OOC narration leak detection gate to the behavioral QA script. A new helper extracts DM player-facing prose from assistant events and matches it against compiled regex patterns. The gate emits FATAL when ≥3 leaks appear in a substantial session, WARN below that threshold, and PASS when clean. The SKILL.md rule is expanded to cover two new forbidden categories. ChangesOOC Narration Leak Gate
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Poem
✨ Finishing Touches📝 Generate docstrings
Comment |
100yenadmin
added a commit
that referenced
this pull request
Jun 17, 2026
…sanitizeNarration (#979) The 2026-06-17 craft audit found the first-person OOC AUTHORING-PREAMBLE family shipping to players verbatim ("Now let me seat <X> as the player character", "Continuity check — let me correct that", "Let me set the order of it", "Here's how round one actually went:", "let me set their advancement through the engine"). PR #972 fixed the leak AT THE SOURCE (DM SKILL.md FICTION-ONLY rule + the deterministic narration_no_ooc_leak gate in qa/assert_behavioral.py::_NARRATION_LEAK). This adds the viewer-arm backstop: a new high-confidence, OOC-only _AUTHORING_PREAMBLE whole-sentence-drop wired into sanitizeNarration's existing _isScaffoldingSentence / _hasScaffolding pass, so any leak that still reaches the chronicle (old transcripts, edge cases) is stripped before the player sees it. Patterns MIRROR _NARRATION_LEAK byte-for-byte (the 5 authoring-preamble arms; the 6th, "inciting incident", is already covered by _CRAFT_JARGON), carrying over its machine-checked FP-hardening: "through the engine" is engine-sense-anchored (literal machinery survives), the seat arm is full-word "as the player character" only, and the round-replay arm is "here's how round <n> … went"-anchored. Additive; legitimate fiction/dialogue is never stripped. Tests: 3 new cases in test_sanitize_narration.py (5 verbatim leaks stripped; inline + multiline surgical strip; legit-fiction FP guards). 13/13 sanitize tests pass; full viewer suite 749 passed (1 pre-existing env failure: pydantic missing in the portrait-gen subprocess, unrelated). Co-authored-by: Eva <arncalso@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A 2026-06-17 adversarial craft audit (10 reads × 2 authored spines, both scored story-craft 4.8) found the 4.8 is inflated (real 4.4-4.7): the prose is BG3-tier, but the scorer blends away a recurring product defect ALL gates missed — OOC DM bookkeeping leaking into player-facing prose (5 verbatim authoring lines shipped to the player in gs-ember-deep). This is the largest gap between the prose the spines write (4.5-4.7) and the felt experience.
Two-part fix (viewer sanitizeNarration defense-in-depth = sibling lane, flagged):
assert_behavioral.pygainsnarration_no_ooc_leak: 0 PASS / 1-2 WARN / ≥3-in-a-substantial-run RED (de-inflates leaky runs). Patterns are high-confidence OOC-only and were adversarially FP-hardened (machine-checked):through the engine→ game-engine-sense-anchored (was matching Gond/Steel-Watch machinery);as the pc→ full-word;your nat…/spine hookdropped from the gate (kept in the SKILL source rule). Real-data: gs-ember-deep → 5 leaks → RED; gs-ledger-deep → 1 → WARN.Changes gate behavior repo-wide — leak-riddled runs now behavioral=RED (intended de-inflation). 8 new tests; 46/46 + fast_gate 222.
Follow-ups: Bless multi-target rider engine bug (next PR, my lane); viewer sanitizeNarration arm (sibling lane).
Summary by CodeRabbit