Skip to content

qa: real known-RED corpus fixture for narration_no_ooc_leak (replaces #977 TODO) - #981

Merged
100yenadmin merged 1 commit into
mainfrom
qa/narration-leak-fixture
Jun 17, 2026
Merged

qa: real known-RED corpus fixture for narration_no_ooc_leak (replaces #977 TODO)#981
100yenadmin merged 1 commit into
mainfrom
qa/narration-leak-fixture

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Jun 17, 2026

Copy link
Copy Markdown
Member

Follow-up to #977

#977 closed the gate-corpus coverage gap for narration_no_ooc_leak with a TODO placeholder (the canonical builder auto-flag). That satisfied the coverage audit but left the anti-goalpost corpus not actually exercising the check's RED path — so a future edit that weakened _NARRATION_LEAK_RE or the n_leak>=3 / dm_text>=MIN_BEATS fatal threshold would slip past the corpus (only test_assert_behavioral.py would catch it).

This replaces the TODO with a faithful minimal known-RED fixture, modeled on the real 2026-06-17 craft audit (#972).

The fixture (qa/gate_corpus/cases/narration_no_ooc_leak/)

6 DM text beats (== MIN_BEATS), 4 leaking the exact OOC patterns the gate bans, 2 clean in-fiction beats:

  • as the player character · continuity check · inciting incident · here's how round one actually went

Isolation (so the per-case corpus test's failed == {expected} holds): a clean roll keeps dice_used green; no companion keeps dm_voices_characters inert; baseline state (day=2, 2 visited, xp>0) passes the world/xp floors; no chat/moves keeps the facade lane inert; 6 (<10) beats + no companion keeps structural_completeness inert.

Verification

Gate run directly on the fixture:

RED: 1 behavioral assertion(s) FAILED.
  [FAIL] narration_no_ooc_leak — 4 player-facing beat(s) leaked OOC craft-scaffolding/bookkeeping/system-vocab — e.g. 'Now let me seat Dal Lightspark as the player character...' [pervasive => RED]

Exactly one FATAL, fully isolated.

$ uv run --directory servers/engine python -m pytest \
    ../../qa/test_behavioral_gate_corpus.py ../../qa/test_root_cause_analyzer.py -q -p no:xdist
39 passed   # was 38 passed + 1 skipped (the TODO is now a real, exercised case)

Builder regeneration is deterministic — zero churn to the other 18 fixtures (diff is the new case dir + the manifest TODO→real-case swap + the generator). assert_behavioral.py untouched.

Summary by CodeRabbit

  • Tests
    • Enhanced test coverage for narration validation scenarios to improve quality assurance.

…laces TODO)

Follow-up to #977, which closed the corpus coverage gap with a TODO placeholder.
This replaces it with a faithful minimal fixture so the anti-goalpost corpus actually
exercises the check's RED path (a future edit that weakens _NARRATION_LEAK_RE or the
n_leak>=3 / dm_text>=MIN_BEATS threshold is now caught here, not just by
test_assert_behavioral.py).

Fixture: 6 DM text beats (== MIN_BEATS), 4 leaking the exact OOC patterns the gate bans
(as-the-player-character / continuity-check / inciting-incident / here's-how-round-N-went),
2 clean in-fiction beats. A clean roll keeps dice_used green; no companion keeps
dm_voices inert; baseline state passes the world/xp floors; no chat/moves keeps the
facade lane inert; 6<10 beats + no companion keeps structural_completeness inert.

Verified ISOLATED: the gate run on the fixture emits exactly
  RED: 1 behavioral assertion(s) FAILED.
  [FAIL] narration_no_ooc_leak — 4 player-facing beat(s) leaked ... [pervasive => RED]
The builder regeneration is deterministic — zero churn to the other 18 fixtures.
Corpus + taxonomy guards: 39 passed (was 38 passed + 1 skipped TODO).
@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e7a78d77-ebc0-4302-9d64-ba3b0e99a592

📥 Commits

Reviewing files that changed from the base of the PR and between 5544ab1 and c75d48f.

📒 Files selected for processing (4)
  • qa/gate_corpus/builder.py
  • qa/gate_corpus/cases/narration_no_ooc_leak/run.jsonl
  • qa/gate_corpus/cases/narration_no_ooc_leak/state.json
  • qa/gate_corpus/manifest.json

📝 Walkthrough

Walkthrough

Promotes the narration_no_ooc_leak behavioral gate from a TODO placeholder to a complete corpus case. builder.py gains a provenance entry, a fixture builder function emitting OOC-leak DM text beats, and a _CASES_SPEC registration. Matching run.jsonl, state.json, and manifest.json entries are added.

Changes

narration_no_ooc_leak corpus case

Layer / File(s) Summary
Fixture builder and corpus registration
qa/gate_corpus/builder.py
REAL_RED_PROVENANCE gains the narration_no_ooc_leak key; case_narration_no_ooc_leak() builds a base roll run and appends DM turns with multiple OOC-leak phrasing patterns; _CASES_SPEC registers the case so the manifest emits expected_red_check: narration_no_ooc_leak.
Generated fixture artifacts and manifest entry
qa/gate_corpus/cases/narration_no_ooc_leak/run.jsonl, qa/gate_corpus/cases/narration_no_ooc_leak/state.json, qa/gate_corpus/manifest.json
run.jsonl records the mcp__engine__roll tool call, its result, and the leaking narration turns; state.json captures the pc1/Dal party snapshot at loc_camp; manifest.json replaces TODO__narration_no_ooc_leak with the fully specified case object.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • electricsheephq/WorldOS#972: Implements the narration_no_ooc_leak behavioral gate in qa/assert_behavioral.py that this PR's corpus case directly targets.
  • electricsheephq/WorldOS#977: Added the TODO__narration_no_ooc_leak placeholder to manifest.json that this PR replaces with the real fixture.

Poem

A rabbit hops past the tavern door,
No OOC whispers shall seep through the floor.
The corpus is fed, the fixture complete,
The gate stands firm — no leaks to defeat!
🐇✨ The TODO is gone, the test is neat!

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The description provides comprehensive context (background from #977, the problem it solved, fixture details, verification steps, and impact), but does not include the required CLA and licensing checkboxes from the template. Add the Licensing/CLA section with required checkboxes and validation checklist from the template to ensure compliance with contribution guidelines.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: replacing a TODO placeholder with a real, known-RED corpus fixture for the narration_no_ooc_leak check, including the reference to the prior issue.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

@100yenadmin
100yenadmin merged commit 3d9d6ac into main Jun 17, 2026
19 checks passed
@100yenadmin
100yenadmin deleted the qa/narration-leak-fixture branch June 17, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant