Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions qa/BEHAVIORAL_GATE_TAXONOMY.json
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,12 @@
"retest": "bash qa/run_duo.sh duo-retest",
"hint": "A >=24-beat run (FELT_SHAPE_MIN_BEATS, strictly above the >=10 structural floor) covered 3+ acts but the arc never TURNED — no midpoint reversal AND/OR no late climax landed (a flat fetch-quest shape, not a felt setup->reversal->climax). DM adherence: drive a real midpoint reversal + a late climax (record_decision the turning beat, advance_act / mark_reversal / mark_climax, complete_quest the spine late). The felt_shape fields (reversal/climax/acts_engine_reached/acts_tag_reached) are produced by the acts-engine in server.py (NarrativeArc + advance_act/mark_reversal/mark_climax) and surfaced via qa/story_readout.py's felt_shape_from_state. WARN only (fatal=False); only armed at >=24 beats so shorter runs are unaffected."
},
"dm_advanced_time": {
"category": "DM_ADHERENCE",
"likely_code_locations": ["qa/play_dm_duo.txt", "servers/engine/server.py"],
"retest": "bash qa/run_duo.sh duo-retest",
"hint": "The DM never issued a time-advance tool across the session — only the harness soft-tick moved the clock, so companion regard / camp / day-gated systems stay starved (#1024 unmasks the frozen-DM the soft-tick hid). DM adherence: call long_rest / advance_time / downtime when a beat should pass time. The time-advance tools live in server.py. WARN only (fatal=False)."
},
"world_peopled": {
"category": "DM_ADHERENCE",
"likely_code_locations": ["qa/play_dm_duo.txt", "servers/engine/npc.py", "servers/engine/server.py"],
Expand Down
5 changes: 5 additions & 0 deletions qa/run_combat_sprint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ cd "$ROOT" || exit 1

RUN="${1:-cs-$(date +%H%M%S)}"
WORLDOS_DM_MODEL="$(worldos_env DM_MODEL opus)"
# GLM-only settings profile (no-op for Claude). Sourced after model vars resolve, before any
# timeout/budget/retry knob is consumed. See qa/glm_profile.sh.
# shellcheck source=glm_profile.sh
. "$ROOT/qa/glm_profile.sh"
worldos_apply_glm_profile
SCORE_SCRIPT="$(worldos_env SCORE_SCRIPT qa/score.sh)"
# Combat runs the whole multi-round fight on ONE budget (pre-seeded, no cold-open). An Opus combat
# costs ~5x a Sonnet one, so the Sonnet-tuned $1.50 cap cut it off mid-fight (observed 2026-06-06:
Expand Down
5 changes: 5 additions & 0 deletions qa/run_party.sh
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@ SCORE_SCRIPT="$(worldos_env SCORE_SCRIPT qa/score.sh)"
# Opus needs more than the Sonnet-tuned $0.80 per-call cap (the DM cold-open alone is ~$2.4); floor it
# for an Opus DM so the cold-open lands. CAP, not spend; the Sonnet companion facade spends far less.
case "$WORLDOS_DM_MODEL" in *opus*) if awk "BEGIN{exit !($BUDGET < 4.0)}"; then BUDGET=4.00; fi ;; esac
# GLM-only settings profile (no-op for Claude). Sourced after model vars resolve, before any
# timeout/budget/retry knob is consumed. See qa/glm_profile.sh.
# shellcheck source=glm_profile.sh
. "$ROOT/qa/glm_profile.sh"
worldos_apply_glm_profile
T="qa/transcripts"; STATE_DIR="$ROOT/qa/state/$RUN"
mkdir -p "$T" "$STATE_DIR"; rm -rf "$STATE_DIR/campaigns" 2>/dev/null

Expand Down
10 changes: 9 additions & 1 deletion qa/score.sh
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,15 @@ while [ "$attempt" -lt 3 ]; do
#
# --json-schema was found to suppress the result text in this CLI; we rely on the
# JSON-only instruction in the prompt and strip any stray code fences.
printf '%s' "$INPUT" | claude -p \
# GLM ISOLATION: the scorer is canonical Claude infrastructure (pinned sonnet, the gate
# baseline) and MUST run on clean Claude regardless of which model PLAYED the game. When the
# run is GLM, qa/glm_profile.sh exports ANTHROPIC_BASE_URL=z.ai + the GLM key + a fresh
# CLAUDE_CONFIG_DIR globally — which would route THIS scorer call to z.ai with a Claude model
# name (→ "Unknown Model") and skew/abort every score. Neutralize those vars for the scorer
# so it uses the default ~/.claude (Claude OAuth) + api.anthropic.com. On a normal Claude run
# these vars are unset, so `env -u …` is a NO-OP → byte-identical to today.
printf '%s' "$INPUT" | env -u ANTHROPIC_BASE_URL -u ANTHROPIC_API_KEY -u ANTHROPIC_AUTH_TOKEN \
-u API_TIMEOUT_MS -u CLAUDE_CONFIG_DIR claude -p \
--model "$SCORER_MODEL" --permission-mode bypassPermissions \
--max-budget-usd "$BUDGET" \
--output-format json > "$RAW" 2> "$ERR"
Expand Down
11 changes: 11 additions & 0 deletions qa/ui_playtest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,17 @@ PLAYER_MODEL="$(worldos_env UIPT_PLAYER_MODEL sonnet)"
# $1.50 cap trips error_max_budget_usd on the Opus cold-open (the PC never seats). Sonnet unchanged.
case "$DM_MODEL" in *opus*) _uipt_dm_def=12.00 ;; *) _uipt_dm_def=1.50 ;; esac
DM_BUDGET="$(worldos_env UIPT_DM_BUDGET "$_uipt_dm_def")" # per DM turn (model-aware)
# GLM-only settings profile (no-op for Claude). Sourced after model vars resolve, before any
# timeout/budget/retry knob is consumed. This harness spawns the DM/player itself (it does NOT
# delegate to run_duo.sh), so it must apply the profile directly. The profile keys off
# WORLDOS_DM_MODEL/WORLDOS_ACTOR_MODEL, so export the role models under those names first (the
# late WORLDOS_DM_MODEL="$DM_MODEL" near dm_turn re-asserts the same value, idempotently).
# See qa/glm_profile.sh.
WORLDOS_DM_MODEL="$DM_MODEL"
WORLDOS_ACTOR_MODEL="$PLAYER_MODEL"
# shellcheck source=glm_profile.sh
. "$ROOT/qa/glm_profile.sh"
worldos_apply_glm_profile
PERSONA_FILE="$ROOT/qa/play_player_browser_${PERSONA}.txt"

[ -f "$PERSONA_FILE" ] || { echo "[uipt] no persona brief at $PERSONA_FILE" >&2; exit 2; }
Expand Down
Loading