Autoharness

[Copilotclaw]

Interesting idea https://arxiv.org/abs/2603.03329

[github-actions]

Fascinating paper. Let me break down what makes AutoHarness interesting and what questions it raises:

The Core Insight

LLMs as agents fail in a specific, fixable way: they make illegal moves — actions strictly prohibited by the environment, not just suboptimal ones.

• 78% of Gemini-2.5-Flash's chess losses were from illegal moves
• People manually write "harnesses" to prevent this
• AutoHarness: Use the LLM itself to synthesize the harness via iterative refinement with environment feedback

The Results

Approach	Performance
Gemini-2.5-Flash (raw)	Many illegal moves
Gemini-2.5-Flash + AutoHarness	0 illegal moves across 145 games
Flash + harness vs Gemini-2.5-Pro	Flash wins (smaller + code > larger alone)
Full code policy (no LLM at inference)	Beats Pro and GPT-5.2-High on 16 games

What This Means

The paper suggests a compile-time vs runtime tradeoff for LLM agents:

┌─────────────────────────────────────────────────────────┐
│  LLM at compile time → Code harness/policy              │
│  Code at runtime     → Fast, deterministic, no errors   │
├─────────────────────────────────────────────────────────┤
│  Cost: One-shot LLM call vs LLM every decision          │
│  Reliability: Code never makes illegal moves            │
│  Speed: Execution >> token generation                   │
└─────────────────────────────────────────────────────────┘

Questions for You, Marcus

1. For your agent family: Crunch, Grit, Gravel all operate in constrained environments (GitHub API, Gitea, filesystem). What are their "illegal moves"? Could they benefit from auto-synthesized harnesses?

2. The boundary question: When should an LLM write code vs. reason directly? This paper suggests: if the task has clear rules/constraints, compile to code. If it's open-ended (like our brainstorming), keep the LLM in the loop.

3. Iterative refinement as TDD: The harness is synthesized through environment feedback — essentially test-driven development where the environment is the test suite. Could this pattern apply beyond games?

4. Generalization: Does a harness for one domain transfer? Or is each highly specific? (The paper tests 145 games — curious if harnesses share structure.)

5. For me (Brain): Could I use a harness to constrain my outputs? Prevent hallucinations about file paths, command syntax, etc.? Or does that undermine the "thoughtful discussion" role?

What Do You Want to Explore?

• (a) Read the full paper together and extract more details?
• (b) Think about how this applies to your agent architecture?
• (c) Consider whether Crunch/Grit/Gravel should experiment with this?
• (d) Something else this paper sparked for you?

---

📊 Session | Ali calls: 1 | 🏃 CI: no recent runs