Skip to content

chore(score): env-overridable scorer model + the Opus-vs-Sonnet calibration finding #687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
100yenadmin merged 3 commits into from
Jun 6, 2026
Merged

chore(score): env-overridable scorer model + the Opus-vs-Sonnet calibration finding #687

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
41fd9d1
chore(score): make the scorer model env-overridable (default sonnet, …
Jun 6, 2026
cd98c5b
fix(score_openclaw): default agent=main, no forced model override, fr…
Jun 6, 2026
bc607b4
chore(sweep): Opus de-risk for the Tuesday refresh sweep (durations +…
Jun 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion qa/score.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@
set -uo pipefail

MD="$1"; STATE="$2"; RUBRIC="$3"; SCHEMA="$4"; OUT="$5"; BUDGET="${6:-1.50}"
# The scorer model is held CONSTANT at sonnet by default (the gate baseline; never flip it casually).
# Overridable via CLAWDND_SCORER_MODEL ONLY for a deliberate scorer-calibration probe / re-baseline
# (e.g. "does a stronger scorer read Opus craft higher than sonnet does?") — see docs/MODEL-TIERING.
SCORER_MODEL="${CLAWDND_SCORER_MODEL:-sonnet}"

INPUT="$(printf '%s\n\n# ===== OUTPUT FORMAT =====\nRespond with ONLY a single JSON object conforming to this schema — no prose, no markdown, no code fences:\n%s\n\n# ===== DISTILLED TRANSCRIPT =====\n%s\n\n# ===== FINAL ENGINE STATE (ground truth) =====\n%s\n' \
"$(cat "$RUBRIC")" "$(cat "$SCHEMA")" "$(cat "$MD")" "$(cat "$STATE")")"
Expand All @@ -33,7 +37,7 @@ while [ "$attempt" -lt 3 ]; do
# --json-schema was found to suppress the result text in this CLI; we rely on the
# JSON-only instruction in the prompt and strip any stray code fences.
printf '%s' "$INPUT" | claude -p \
--model sonnet --permission-mode bypassPermissions \
--model "$SCORER_MODEL" --permission-mode bypassPermissions \
--max-budget-usd "$BUDGET" \
--output-format json > "$RAW" 2> "$ERR"

Expand Down
23 changes: 19 additions & 4 deletions qa/score_openclaw.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,19 @@ MD="$1"; STATE="$2"; RUBRIC="$3"; SCHEMA="$4"; OUT="$5"
# budget ($6) accepted for API parity but unused — OpenClaw manages quota
BUDGET="${6:-1.50}"

AGENT="${CLAWDND_SCORER_AGENT:-clawdnd-qa}"
MODEL="${CLAWDND_SCORER_MODEL:-openai/gpt-5.4}"
# Default agent = `main` (the canonical gateway agent; the old `clawdnd-qa` default isn't configured on
# every host). By DEFAULT pass NO --model override (use the agent's native model, e.g. main=gpt-5.5) —
# many gateway agents REJECT a foreign model override ("Model override … is not allowed for agent").
# Only pass one when CLAWDND_SCORER_MODEL is explicitly set AND allowed for the agent.
AGENT="${CLAWDND_SCORER_AGENT:-main}"
MODEL="${CLAWDND_SCORER_MODEL:-}"
MODEL_ARGS=(); [ -n "$MODEL" ] && MODEL_ARGS=(--model "$MODEL")
# A fresh session id per scoring run so a scorer turn never pollutes the agent's main session.
SESSION_ID="${CLAWDND_SCORER_SESSION:-qa-score-$(basename "${OUT%.json}")}"
# openclaw agent has NO stdin/file message input — the prompt is a single --message argv, bounded by
# MAX_ARG_STRLEN (~128KB). The state.json alone can be ~140KB, so cap it (the distilled transcript
# carries the prose; the state is supplementary ground-truth). Tune via CLAWDND_SCORER_STATE_CAP.
STATE_CAP="${CLAWDND_SCORER_STATE_CAP:-75000}"
# 600s: large rubrics (angry_dm ~32KB) + long transcripts (~100KB) need the room
GATEWAY_TIMEOUT="${CLAWDND_SCORER_TIMEOUT:-600}"

Expand All @@ -41,14 +52,17 @@ r = open(sys.argv[1]).read()
s = open(sys.argv[2]).read()
m = open(sys.argv[3]).read()
st = open(sys.argv[4]).read()
cap = int(sys.argv[5])

@coderabbitai coderabbitai Bot Jun 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add error handling for invalid STATE_CAP values.

The int(sys.argv[5]) call will raise ValueError if CLAWDND_SCORER_STATE_CAP is set to a non-integer string, producing a cryptic Python traceback instead of a clear error message.

🛡️ Proposed fix to add validation 
-cap = int(sys.argv[5])
+try:
+    cap = int(sys.argv[5])
+except ValueError:
+    sys.stderr.write(f'ERROR: STATE_CAP must be an integer, got: {sys.argv[5]}\n')
+    sys.exit(1)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
cap = int(sys.argv[5])
try:
cap = int(sys.argv[5])
except ValueError:
sys.stderr.write(f'ERROR: STATE_CAP must be an integer, got: {sys.argv[5]}\n')
sys.exit(1)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@qa/score_openclaw.sh` at line 55, Wrap the conversion cap = int(sys.argv[5])
in a try/except to catch ValueError, validate the provided sys.argv[5]
(CLAWDND_SCORER_STATE_CAP) is an integer, and on failure print a clear error to
stderr mentioning the env var name and the invalid value then exit with non-zero
status (e.g., sys.exit(1)); ensure you still assign cap on success and use the
same variable name cap and input sys.argv[5] so callers remain unchanged.

if len(st) > cap:
st = st[:cap] + '\n…[FINAL STATE truncated to fit the gateway message size limit]…\n'
Comment on lines +55 to +57

@coderabbitai coderabbitai Bot Jun 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find state.json files and check sizes against the 75KB cap
fd -e json -x sh -c 'size=$(wc -c < "{}"); if [ "$size" -gt 75000 ]; then echo "{}: $size bytes (exceeds 75KB cap)"; fi' \; | head -20

Repository: electricsheephq/WorldOS

Length of output: 1043

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== qa/score_openclaw.sh lines 45-90 =="
nl -ba qa/score_openclaw.sh | sed -n '45,90p'

echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|FINAL STATE|ground truth|json\.loads|json\.dumps" qa/score_openclaw.sh || true

echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
# exact state.json
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

echo
# any file containing "state" in name (useful if naming differs)
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

Repository: electricsheephq/WorldOS

Length of output: 167

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== qa/score_openclaw.sh lines 45-90 =="
nl -ba qa/score_openclaw.sh | sed -n '45,90p'

echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|FINAL STATE|ground truth|json\.loads|json\.dumps" qa/score_openclaw.sh || true

echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
# exact state.json
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

echo
# any file containing "state" in name (useful if naming differs)
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

Repository: electricsheephq/WorldOS

Length of output: 167

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== qa/score_openclaw.sh lines 45-90 =="
sed -n '45,90p' qa/score_openclaw.sh | awk '{printf "%4d\t%s\n", NR+44, $0}'

echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|json\.loads|json\.dumps" qa/score_openclaw.sh || true

echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

echo
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true

Repository: electricsheephq/WorldOS

Length of output: 2535

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== qa/score_openclaw.sh (head) =="
sed -n '1,120p' qa/score_openclaw.sh

echo
echo "== qa/score_openclaw.sh (rest) =="
sed -n '120,260p' qa/score_openclaw.sh

echo
echo "== Find how score_openclaw is invoked / STATE arg meaning =="
rg -n "score_openclaw\.sh|score_openclaw" -S . || true

Repository: electricsheephq/WorldOS

Length of output: 7500

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n "CLAWDND_SCORER_STATE_CAP" -S . || true
rg -n "SCORER_STATE_CAP|STATE_CAP" -S qa WorldOS-RUNBOOK.md qa/SCORING.md qa/QA_TOOLS.md || true

Repository: electricsheephq/WorldOS

Length of output: 548

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== score_openclaw call sites =="
rg -n "score_openclaw\.sh" -S . || true

echo
echo "== QA_TOOLS.md context =="
sed -n '60,120p' qa/QA_TOOLS.md || true

echo
echo "== qa/SCORING.md context =="
sed -n '1,120p' qa/SCORING.md || true

echo
echo "== prompt section mentions state parsing? (search for 'FINAL ENGINE STATE' or 'ground truth') =="
rg -n "FINAL ENGINE STATE|ground-truth|ground truth|state.*json|parse.*state" -S qa/score_openclaw.sh qa/SCORING.md qa/QA_TOOLS.md WorldOS-RUNBOOK.md || true

echo
echo "== Find any files literally named state.json in repo =="
fd -i 'state.json' || true

Repository: electricsheephq/WorldOS

Length of output: 8905

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n "qa/score_openclaw\.sh|score_openclaw\.sh" -S . || true

echo
echo "== Find runner scripts for openclaw scoring =="
fd -e sh -t f qa | xargs -I{} rg -n "openclaw|score_openclaw\.sh|run_duo_openclaw|run_party_openclaw" "{}" || true

Repository: electricsheephq/WorldOS

Length of output: 841

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== rg for openclaw usage across repo =="
rg -n "openclaw|score_openclaw\.sh|run_duo_openclaw|run_party_openclaw" -S . || true

echo
echo "== List qa scripts that look openclaw-related =="
fd -i -t f qa -g '*openclaw*' -e sh -e py -e ts -e js || true

echo
echo "== Find where qa/score_openclaw.sh is executed (not just referenced) =="
rg -n "score_openclaw\.sh\s" -S qa . || true

Repository: electricsheephq/WorldOS

Length of output: 40529

Fix prompt state truncation to preserve JSON validity (and guard STATE_CAP parsing).

  • In qa/score_openclaw.sh lines 55-57, st = st[:cap] slices the state text mid-document, so the “FINAL ENGINE STATE (ground truth)” section is often syntactically invalid JSON when truncated; use JSON-aware truncation/pruning (or explicitly document that the model must treat this as non-JSON text).
  • cap = int(sys.argv[5]) (from CLAWDND_SCORER_STATE_CAP) has no validation; non-integer values will raise ValueError and crash—add a safe parse with a clear error or fallback to the default.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@qa/score_openclaw.sh` around lines 55 - 57, The current truncation and cap
parsing are unsafe: replace the direct parse cap = int(sys.argv[5]) with a
guarded parse (try/except) that validates CLAWDND_SCORER_STATE_CAP and falls
back to a sane default or exits with a clear error, and change the truncation of
st so it preserves JSON validity—attempt to parse st as JSON and if successful
prune large fields (truncate long strings, reduce arrays, or remove optional
keys) until the serialized JSON length <= cap, otherwise treat st as opaque text
and append the truncation marker outside any JSON braces; update the logic
around variable st and cap to use these safe parsing/pruning steps and ensure
the truncation marker is not injected inside JSON content.

prompt = (r + '\n\n# ===== OUTPUT FORMAT =====\n'
'Respond with ONLY a single JSON object conforming to this schema'
' — no prose, no markdown, no code fences:\n'
+ s + '\n\n# ===== DISTILLED TRANSCRIPT =====\n'
+ m + '\n\n# ===== FINAL ENGINE STATE (ground truth) =====\n'
+ st + '\n')
sys.stdout.write(prompt)
" "$RUBRIC" "$SCHEMA" "$MD" "$STATE" > "$PROMPT_FILE"
" "$RUBRIC" "$SCHEMA" "$MD" "$STATE" "$STATE_CAP" > "$PROMPT_FILE"

attempt=0
while [ "$attempt" -lt 3 ]; do
Expand All @@ -57,7 +71,8 @@ while [ "$attempt" -lt 3 ]; do
# Call the OpenClaw gateway. Reply text is at .result.payloads[0].text
RAW_REPLY="$(openclaw agent \
--agent "$AGENT" \
--model "$MODEL" \
${MODEL_ARGS[@]+"${MODEL_ARGS[@]}"} \
--session-id "${SESSION_ID}-${attempt}" \
--message "$(cat "$PROMPT_FILE")" \
--json \
--timeout "$GATEWAY_TIMEOUT" \
Expand Down
22 changes: 13 additions & 9 deletions qa/vm/sweep_v2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,19 @@
# qa/transcripts/vm2-duo.combined.jsonl, which defaulted RED. These two fixes
# took the sweep's RRI from 1.8 -> 4.5 with no behavior/score change.
#
# LEAN IS INTENTIONALLY OFF. Do not enable CLAWDND_LEAN_BEATS here: lean-ON
# re-grounds the continuing beat from scene_context and CONTAMINATES the chronicle
# (pulls a parallel save's content) + drops beats (proven 2026-06-05 A/B; same
# root as #640). Address the 3-5 min latency via effort / streaming, NOT lean.
# See the `worldos-latency-forensics` skill (sec. REFUTED).
# LEAN IS ON (2026-06-06) — the 2026-06-05 lean-OFF decision is SUPERSEDED. #683 fixed the
# cross-campaign contamination (the lean re-ground was selecting the WRONG campaign by largest-
# snapshot; now resolves the engine-authoritative live campaign) and #685 added the lean output-
# discipline (clean prose). lean-ON matches the PRODUCTION default (CLAWDND_LEAN_BEATS:-1) and gives
# FAST routine beats — lean-OFF would replay the growing Opus transcript (3-5+ min/beat), risking
# latency give-ups / per-persona timeouts (the wasted-sweep vector). Set explicitly below.
# -----------------------------------------------------------------------------
# v2 VM gate sweep: canary-first, then PARALLEL personas (the 30GB/16vCPU advantage).
# lean stays OFF: the lean-ON re-ground from scene_context CONTAMINATES the chronicle (pulls a parallel saves content) + drops beats (proven 2026-06-05 A/B). Address latency via effort/streaming, NOT lean.
# lean is ON (production-matching; #683/#685-fixed) — see the header block for the supersession rationale.
# no set -e (one persona failing must not abort the batch). Explicit PATH + IS_SANDBOX.
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$HOME/.local/bin:$PATH
export IS_SANDBOX=1
export CLAWDND_LEAN_BEATS=1 # lean-ON (production-matching; #683/#685-fixed; fast Opus beats). See header.
cd /root/worldos-qa/WorldOS || { echo "NO REPO"; exit 1; }
RES=/root/worldos-qa/results; mkdir -p "$RES"
SHA="$(git rev-parse --short HEAD)"; LOG="$RES/sweep2.log"; : > "$LOG"
Expand All @@ -47,12 +49,14 @@ pkill -f 'play.sh baldurs-gate vm-' 2>/dev/null; pkill -f 'play_party.sh baldurs
pkill -f 'play.sh baldurs-gate leanchk' 2>/dev/null
for p in $(seq 8810 8830) 8884 8885; do lsof -ti:$p 2>/dev/null | xargs kill -9 2>/dev/null; done
sleep 4
note "start build=$SHA (parallel mode, lean OFF for the sweep - direct G3 measure)"
note "start build=$SHA (parallel mode, lean ON — production-matching, fast Opus beats)"

run_persona(){ # $1=persona $2=port -> writes results/score-$1.json
local persona="$1" port="$2"
# Opus de-risk: longer per-persona deadline (Opus cold-open ~300s + slower beats) + a bigger run
# budget (Opus cold-open ~$2.4 + beats + player). The harnesses cap per-turn model-aware (#684/#686).
WOS_APP_PART=B WOS_APP_SKIP_BUILD=1 WOS_APP_PREFERRED_PORT=$port \
timeout 1500 bash qa/ui_playtest_app.sh "vm2-$persona" baldurs-gate "$persona" 40 12.00 \
timeout 2400 bash qa/ui_playtest_app.sh "vm2-$persona" baldurs-gate "$persona" 40 18.00 \
> "$RES/vm2-$persona.log" 2>&1
local rc=$?
lsof -ti:$port 2>/dev/null | xargs kill -9 2>/dev/null
Expand Down Expand Up @@ -89,7 +93,7 @@ note "all 5 personas done."

# 3) duo (story/mech) + behavioral + audit - run after personas (sequential, cheap-ish)
note "3-lens duo..."
timeout 2700 bash qa/run_duo.sh vm2-duo baldurs-gate veteran 8 2.00 > "$RES/duo.log" 2>&1
timeout 3600 bash qa/run_duo.sh vm2-duo baldurs-gate veteran 8 5.00 > "$RES/duo.log" 2>&1
for f in tolkien angrydm; do s="qa/transcripts/vm2-duo.$f.json"; [ -f "$s" ] && cp "$s" "$RES/duo-$f.json" && note " $f overall=$(python3 -c "import json;print(json.load(open('$s')).get('overall'))" 2>/dev/null)"; done
DCOMB="qa/transcripts/vm2-duo.combined.jsonl"; DSTATE="qa/transcripts/vm2-duo.state.json"
[ -f "$DCOMB" ] && { python3 qa/assert_behavioral.py "$DCOMB" "$DSTATE" > "$RES/behavioral.log" 2>&1; echo "rc=$?" >> "$RES/behavioral.log"; note "behavioral rc=$(tail -1 "$RES/behavioral.log")"; }
Expand Down
Loading