Skip to content

qa: sync gate guards (root-cause taxonomy + corpus manifest) with structural_completeness + narration_no_ooc_leak - #977

Merged
100yenadmin merged 3 commits into
mainfrom
qa/taxonomy-sync-checks
Jun 17, 2026
Merged

qa: sync gate guards (root-cause taxonomy + corpus manifest) with structural_completeness + narration_no_ooc_leak#977
100yenadmin merged 3 commits into
mainfrom
qa/taxonomy-sync-checks

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Jun 17, 2026

Copy link
Copy Markdown
Member

Problem

Two QA guard tests were RED on main (the CI / qa-release-gate-tests job has been failing on main for several commits). Both stem from one root cause: the gate checks structural_completeness and narration_no_ooc_leak were added to qa/assert_behavioral.py without updating the guard metadata that's supposed to track the gate.

  1. test_root_cause_analyzer.py::test_taxonomy_matches_real_gate_check_names — the root-cause taxonomy (qa/BEHAVIORAL_GATE_TAXONOMY.json) was missing both checks.
  2. test_behavioral_gate_corpus.py::test_manifest_covers_every_fatal_check — the gate-corpus manifest had no fixture and no TODO entry for narration_no_ooc_leak (a FATAL check: chk(... fatal=_leak_red)). structural_completeness already had a fixture, so only the taxonomy was missing it.

Both were pre-existing on main and independent of each other; the corpus failure does not read the taxonomy at all.

Fix (guard metadata only — assert_behavioral.py untouched)

Taxonomy (qa/BEHAVIORAL_GATE_TAXONOMY.json) — added both entries:

check category rationale
structural_completeness ENGINE_INVARIANT FATAL duo-lane check that a ≥10-beat companion session engaged a core relationship/quest system. Reads engine-written state — structurally parallel to party_traveled / world_advanced_time. Locations → companion.py / server.py / questgen.py.
narration_no_ooc_leak DM_ADHERENCE Player-facing OOC craft-scaffolding leak gate (0 pass / 1-2 WARN / ≥3 RED). About the DM's narrated prose; remediation → FICTION-ONLY mandate in skills/dungeon-master/SKILL.md + duo DM prompt; _NARRATION_LEAK_RE as the false-positive tuning point.

Corpus manifest (qa/gate_corpus/manifest.json) — ran the canonical sync python qa/gate_corpus/builder.py, which deterministically regenerated all 18 fixtures (byte-identical, zero churn) and auto-added a TODO__narration_no_ooc_leak entry (a faithful minimal OOC-leak fixture is deferred work, flagged with a reason).

Test

$ uv run --directory servers/engine python -m pytest \
    ../../qa/test_behavioral_gate_corpus.py ../../qa/test_root_cause_analyzer.py -q -p no:xdist
38 passed, 1 skipped

No behavior change to any gate; only the analyzer's RED→actionable-location mapping and the corpus coverage audit gain the two checks.

Summary by CodeRabbit

  • Tests
    • Added new automated QA checks to enforce narrative consistency (preventing out-of-character information leakage) and structural completeness.
    • Introduced a new regression-corpus case for the narrative consistency check, currently marked as pending until the minimal required fixture content is available.

…_no_ooc_leak checks

The taxonomy-drift guard test (qa/test_root_cause_analyzer.py::
test_taxonomy_matches_real_gate_check_names) was RED on main: two chk() names
added to qa/assert_behavioral.py were never mirrored into the root-cause taxonomy.

- structural_completeness (ENGINE_INVARIANT) — FATAL duo-lane check that a >=10-beat
  companion session engaged a core relationship/quest system; mapped to the
  companion/server/questgen engine surfaces, consistent with the sibling
  party_traveled / world_advanced_time structural checks.
- narration_no_ooc_leak (DM_ADHERENCE, #972) — player-facing OOC-leak gate; mapped to
  the FICTION-ONLY mandate in skills/dungeon-master/SKILL.md + the duo DM prompt, with
  the leak-regex in assert_behavioral.py as the false-positive tuning point.

Pure metadata sync; assert_behavioral.py untouched. Test now 19/19 green.
@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c27ba28e-e8b6-49c5-b4d6-67342e93f1e0

📥 Commits

Reviewing files that changed from the base of the PR and between abb0df1 and a921a19.

📒 Files selected for processing (1)
  • qa/BEHAVIORAL_GATE_TAXONOMY.json
🚧 Files skipped from review as they are similar to previous changes (1)
  • qa/BEHAVIORAL_GATE_TAXONOMY.json

📝 Walkthrough

Walkthrough

Two new check entries are appended to qa/BEHAVIORAL_GATE_TAXONOMY.json: narration_no_ooc_leak mapped to the DM_ADHERENCE category and structural_completeness mapped to ENGINE_INVARIANT. Each entry includes candidate code locations, a duo retest command, and a descriptive hint. A corresponding todo corpus case is added to qa/gate_corpus/manifest.json for the narration check, flagged pending fixture construction.

Changes

Behavioral Gate Taxonomy Additions

Layer / File(s) Summary
New taxonomy check entries
qa/BEHAVIORAL_GATE_TAXONOMY.json
Adds narration_no_ooc_leak (DM_ADHERENCE) with candidate locations in DM skill doc, DM duo play script, and qa/assert_behavioral.py, and structural_completeness (ENGINE_INVARIANT) with candidate locations in companion, server, and questgen code paths; both entries include exact duo retest commands and threshold hints.
Corpus test case registry
qa/gate_corpus/manifest.json
Adds a TODO__narration_no_ooc_leak case entry marked todo: true with a reason noting that the minimal fixture for the narration check has not yet been constructed.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~4 minutes

Possibly related PRs

  • electricsheephq/WorldOS#972: Implements the narration_no_ooc_leak behavioral gate and its tests in qa/assert_behavioral.py, directly corresponding to the taxonomy entry added in this PR.

Poem

🐇 Two new gates now guard the realm,
No out-of-character ghosts at the helm!
The DM speaks only in-fiction true,
Quests and companions complete their due.
Hoppity-hop, the taxonomy grew! 🗂️

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and concisely describes the main change: syncing gate metadata (taxonomy and corpus) with two new gate checks. It is specific, clear, and directly reflects the changeset.
Description check ✅ Passed The description comprehensively covers the problem, fix, and validation. It includes problem statement, detailed fix explanation with a table, test results, and explicitly states no behavior changes. The Licensing/CLA section is missing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

…try)

test_behavioral_gate_corpus.py::test_manifest_covers_every_fatal_check was RED on
main (independent of the taxonomy drift): narration_no_ooc_leak is a FATAL gate
check (chk(... fatal=_leak_red)) with no corpus fixture and no TODO manifest entry,
so the anti-goalpost coverage audit failed.

Ran the canonical sync — python qa/gate_corpus/builder.py — which deterministically
regenerated all 18 fixtures (byte-identical, zero churn) and auto-added the
TODO__narration_no_ooc_leak manifest entry (a faithful minimal OOC-leak fixture is
deferred work). Both guard suites now green: test_behavioral_gate_corpus.py +
test_root_cause_analyzer.py = 38 passed, 1 skipped.
@100yenadmin 100yenadmin changed the title qa: sync root-cause taxonomy with structural_completeness + narration_no_ooc_leak qa: sync gate guards (root-cause taxonomy + corpus manifest) with structural_completeness + narration_no_ooc_leak Jun 17, 2026
…w fix)

A 3-agent verification of this PR flagged two MAJOR inaccuracies in the
structural_completeness location hints that would misdirect an agent reading a RED gate:

- questgen.py was named for the 'quest-resolution/evolution path' but holds ZERO such
  code (it is the quest-SEED-generation layer; 0 hits for evolves_to/complete_quest).
  The rule-of-three evolution (_maybe_schedule_quest_evolution, evolves_to write) lives
  in server.py. Replaced questgen.py with models.py (the field defs).
- companion.py was credited with 'the approval/attitude write' but is a deliberately
  PURE module with no campaign I/O — every attitude_value/adjust_attitude write is in
  server.py. Kept companion.py but reworded the hint to its real role: it only surfaces
  approval CAUSES (approval_tags) for the DM to apply.

Also disambiguated the 10-beat gate trip (STRUCTURAL_MIN_BEATS) from the >=24-beat
authored-campaign validation recommendation. No key/category change; guards still
38 passed, 1 skipped.
@100yenadmin
100yenadmin merged commit 5544ab1 into main Jun 17, 2026
19 checks passed
@100yenadmin
100yenadmin deleted the qa/taxonomy-sync-checks branch June 17, 2026 07:13
100yenadmin added a commit that referenced this pull request Jun 17, 2026
…977 TODO)

Replaces the TODO placeholder #977 added with a faithful minimal known-RED fixture so the anti-goalpost corpus actually exercises narration_no_ooc_leak's RED path.

6 DM text beats (== MIN_BEATS), 4 leaking the exact OOC patterns the gate bans; isolated so the gate emits exactly `RED: 1 ... [FAIL] narration_no_ooc_leak ... [pervasive => RED]`. Builder regeneration deterministic (zero churn to the other 18 fixtures); assert_behavioral.py untouched. Guards: 39 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant