chore(score): env-overridable scorer model + the Opus-vs-Sonnet calibration finding - #687
Conversation
…gate-constant) Adds CLAWDND_SCORER_MODEL (default sonnet) so the gate scorer stays constant by default, but a deliberate scorer-calibration probe can re-score with a stronger judge. Motivated by the analytical-Sonnet (story 4.0) vs lived-persona (~4.5 'best AI fiction') gap on Opus craft.
|
Lost in the diff? Review this PR in Change Stack to follow the change map from intent to exact ranges. Warning Review limit reached
More reviews will be available in 1 hour, 7 minutes, and 39 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughQA scoring scripts: ChangesScorer + OpenClaw gateway updates
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
…esh session, state cap The gateway scorer was broken on the VM: default agent clawdnd-qa isn't configured (main is, model gpt-5.5), --model openai/gpt-5.4 is REJECTED for main, and the single --message argv tripped E2BIG (state.json ~137KB > MAX_ARG_STRLEN ~128KB). Fixes: default agent=main, pass --model only when explicitly set+allowed, fresh --session-id per run (no main-session pollution), and cap the state (CLAWDND_SCORER_STATE_CAP=75000; transcript carries the prose). Enables the GPT-5.5 3rd judge + the credit-saving gateway scorer the owner asked for.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@qa/score_openclaw.sh`:
- Line 55: Wrap the conversion cap = int(sys.argv[5]) in a try/except to catch
ValueError, validate the provided sys.argv[5] (CLAWDND_SCORER_STATE_CAP) is an
integer, and on failure print a clear error to stderr mentioning the env var
name and the invalid value then exit with non-zero status (e.g., sys.exit(1));
ensure you still assign cap on success and use the same variable name cap and
input sys.argv[5] so callers remain unchanged.
- Around line 55-57: The current truncation and cap parsing are unsafe: replace
the direct parse cap = int(sys.argv[5]) with a guarded parse (try/except) that
validates CLAWDND_SCORER_STATE_CAP and falls back to a sane default or exits
with a clear error, and change the truncation of st so it preserves JSON
validity—attempt to parse st as JSON and if successful prune large fields
(truncate long strings, reduce arrays, or remove optional keys) until the
serialized JSON length <= cap, otherwise treat st as opaque text and append the
truncation marker outside any JSON braces; update the logic around variable st
and cap to use these safe parsing/pruning steps and ensure the truncation marker
is not injected inside JSON content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: dd1fe909-7b55-4e4a-855b-7a77accebabd
📒 Files selected for processing (1)
qa/score_openclaw.sh
| s = open(sys.argv[2]).read() | ||
| m = open(sys.argv[3]).read() | ||
| st = open(sys.argv[4]).read() | ||
| cap = int(sys.argv[5]) |
There was a problem hiding this comment.
Add error handling for invalid STATE_CAP values.
The int(sys.argv[5]) call will raise ValueError if CLAWDND_SCORER_STATE_CAP is set to a non-integer string, producing a cryptic Python traceback instead of a clear error message.
🛡️ Proposed fix to add validation
-cap = int(sys.argv[5])
+try:
+ cap = int(sys.argv[5])
+except ValueError:
+ sys.stderr.write(f'ERROR: STATE_CAP must be an integer, got: {sys.argv[5]}\n')
+ sys.exit(1)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| cap = int(sys.argv[5]) | |
| try: | |
| cap = int(sys.argv[5]) | |
| except ValueError: | |
| sys.stderr.write(f'ERROR: STATE_CAP must be an integer, got: {sys.argv[5]}\n') | |
| sys.exit(1) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@qa/score_openclaw.sh` at line 55, Wrap the conversion cap = int(sys.argv[5])
in a try/except to catch ValueError, validate the provided sys.argv[5]
(CLAWDND_SCORER_STATE_CAP) is an integer, and on failure print a clear error to
stderr mentioning the env var name and the invalid value then exit with non-zero
status (e.g., sys.exit(1)); ensure you still assign cap on success and use the
same variable name cap and input sys.argv[5] so callers remain unchanged.
| cap = int(sys.argv[5]) | ||
| if len(st) > cap: | ||
| st = st[:cap] + '\n…[FINAL STATE truncated to fit the gateway message size limit]…\n' |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find state.json files and check sizes against the 75KB cap
fd -e json -x sh -c 'size=$(wc -c < "{}"); if [ "$size" -gt 75000 ]; then echo "{}: $size bytes (exceeds 75KB cap)"; fi' \; | head -20Repository: electricsheephq/WorldOS
Length of output: 1043
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== qa/score_openclaw.sh lines 45-90 =="
nl -ba qa/score_openclaw.sh | sed -n '45,90p'
echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|FINAL STATE|ground truth|json\.loads|json\.dumps" qa/score_openclaw.sh || true
echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
# exact state.json
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true
echo
# any file containing "state" in name (useful if naming differs)
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || trueRepository: electricsheephq/WorldOS
Length of output: 167
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== qa/score_openclaw.sh lines 45-90 =="
nl -ba qa/score_openclaw.sh | sed -n '45,90p'
echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|FINAL STATE|ground truth|json\.loads|json\.dumps" qa/score_openclaw.sh || true
echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
# exact state.json
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true
echo
# any file containing "state" in name (useful if naming differs)
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || trueRepository: electricsheephq/WorldOS
Length of output: 167
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== qa/score_openclaw.sh lines 45-90 =="
sed -n '45,90p' qa/score_openclaw.sh | awk '{printf "%4d\t%s\n", NR+44, $0}'
echo
echo "== Search in qa/score_openclaw.sh for key strings =="
rg -n "sys\.argv\[5\]|CLAWDND_SCORER_STATE_CAP|st\s*\[:cap\]|json\.loads|json\.dumps" qa/score_openclaw.sh || true
echo
echo "== Find files that match state.json exactly (and *state*.json) and check size > 75KB =="
fd -i 'state.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || true
echo
fd -i '*state*.json' -e json -t f --exec sh -c 's=$(wc -c < "{}"); if [ "$s" -gt 75000 ]; then echo "{}: ${s} bytes"; fi' \; 2>/dev/null | head -50 || trueRepository: electricsheephq/WorldOS
Length of output: 2535
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== qa/score_openclaw.sh (head) =="
sed -n '1,120p' qa/score_openclaw.sh
echo
echo "== qa/score_openclaw.sh (rest) =="
sed -n '120,260p' qa/score_openclaw.sh
echo
echo "== Find how score_openclaw is invoked / STATE arg meaning =="
rg -n "score_openclaw\.sh|score_openclaw" -S . || trueRepository: electricsheephq/WorldOS
Length of output: 7500
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "CLAWDND_SCORER_STATE_CAP" -S . || true
rg -n "SCORER_STATE_CAP|STATE_CAP" -S qa WorldOS-RUNBOOK.md qa/SCORING.md qa/QA_TOOLS.md || trueRepository: electricsheephq/WorldOS
Length of output: 548
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== score_openclaw call sites =="
rg -n "score_openclaw\.sh" -S . || true
echo
echo "== QA_TOOLS.md context =="
sed -n '60,120p' qa/QA_TOOLS.md || true
echo
echo "== qa/SCORING.md context =="
sed -n '1,120p' qa/SCORING.md || true
echo
echo "== prompt section mentions state parsing? (search for 'FINAL ENGINE STATE' or 'ground truth') =="
rg -n "FINAL ENGINE STATE|ground-truth|ground truth|state.*json|parse.*state" -S qa/score_openclaw.sh qa/SCORING.md qa/QA_TOOLS.md WorldOS-RUNBOOK.md || true
echo
echo "== Find any files literally named state.json in repo =="
fd -i 'state.json' || trueRepository: electricsheephq/WorldOS
Length of output: 8905
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "qa/score_openclaw\.sh|score_openclaw\.sh" -S . || true
echo
echo "== Find runner scripts for openclaw scoring =="
fd -e sh -t f qa | xargs -I{} rg -n "openclaw|score_openclaw\.sh|run_duo_openclaw|run_party_openclaw" "{}" || trueRepository: electricsheephq/WorldOS
Length of output: 841
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== rg for openclaw usage across repo =="
rg -n "openclaw|score_openclaw\.sh|run_duo_openclaw|run_party_openclaw" -S . || true
echo
echo "== List qa scripts that look openclaw-related =="
fd -i -t f qa -g '*openclaw*' -e sh -e py -e ts -e js || true
echo
echo "== Find where qa/score_openclaw.sh is executed (not just referenced) =="
rg -n "score_openclaw\.sh\s" -S qa . || trueRepository: electricsheephq/WorldOS
Length of output: 40529
Fix prompt state truncation to preserve JSON validity (and guard STATE_CAP parsing).
- In
qa/score_openclaw.shlines 55-57,st = st[:cap]slices the state text mid-document, so the “FINAL ENGINE STATE (ground truth)” section is often syntactically invalid JSON when truncated; use JSON-aware truncation/pruning (or explicitly document that the model must treat this as non-JSON text). cap = int(sys.argv[5])(fromCLAWDND_SCORER_STATE_CAP) has no validation; non-integer values will raiseValueErrorand crash—add a safe parse with a clear error or fallback to the default.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@qa/score_openclaw.sh` around lines 55 - 57, The current truncation and cap
parsing are unsafe: replace the direct parse cap = int(sys.argv[5]) with a
guarded parse (try/except) that validates CLAWDND_SCORER_STATE_CAP and falls
back to a sane default or exits with a clear error, and change the truncation of
st so it preserves JSON validity—attempt to parse st as JSON and if successful
prune large fields (truncate long strings, reduce arrays, or remove optional
keys) until the serialized JSON length <= cap, otherwise treat st as opaque text
and append the truncation marker outside any JSON braces; update the logic
around variable st and cap to use these safe parsing/pruning steps and ensure
the truncation marker is not injected inside JSON content.
… budgets + lean-ON) For the scheduled Tue-4am full sweep on the credit refresh: per-persona timeout 1500->2400s + run budget $12->$18 (Opus cold-open ~$2.4 + slower beats); duo timeout 2700->3600s + budget $2->$5. Flip lean ON (was a stale 'intentionally OFF' from the 2026-06-05 finding; #683 fixed the cross-campaign contamination + #685 the output-discipline) — lean-ON matches production + gives fast Opus beats so the sweep completes without latency give-ups/timeouts (the wasted-run vector).
1 participant
Adds
CLAWDND_SCORER_MODELtoqa/score.sh(default sonnet — the gate scorer stays constant by default; never flipped casually). This enabled a scorer-calibration probe that produced a pivotal finding.Finding: the Sonnet gate scorer UNDER-READS Opus story craft
Re-scored 3 existing transcripts with both judges:
Not a self-favoring bias — the Opus judge reads story higher but mech lower, so it isn't inflating Opus output; it's a more discerning judge. On subjective prose craft the Sonnet scorer flatlines at uniform 4s; on objective rules-fidelity both agree (Opus stricter). The Opus rationale is specific and critical ("Near BG3-tier… the single biggest lever is restraint — stop narrating the hero's interior realizations"), corroborated by the GUI persona's lived ~4.5.
Implication
Caveat: a fully rigorous story-scorer recalibration wants a 3rd independent judge (GPT-5.4) or a human anchor — the #643 scorer-audit's job. This PR is the tooling + the evidence.
Summary by CodeRabbit
Chores
Bug Fixes