Skip to content

Add OpenCode as a third execution harness - #218

Open
chenmingtang830 wants to merge 6 commits into
codex/transcript-decoder-seamfrom
codex/opencode-harness
Open

Add OpenCode as a third execution harness#218
chenmingtang830 wants to merge 6 commits into
codex/transcript-decoder-seamfrom
codex/opencode-harness

Conversation

@chenmingtang830

@chenmingtang830 chenmingtang830 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • add OpenCode 1.18.3+ as a third open-source core harness for API, CLI, and SDK cells
  • require an explicit provider/model route and scope only declared provider and target environment values
  • decode OpenCode JSONL through the stable event seam from Refactor transcript decoding behind stable events #217, including normalized tool counts and token/cache metrics
  • fail OpenCode MCP cells closed as unsupported before credentials, health checks, provisioning, or invocation
  • preserve the historical ax.normalized-result/v1 enum while one-cell records carry unsupported-surface

Isolation and evidence

  • run with --pure and --auto from a disposable cwd outside the checkout, with private HOME/XDG roots, no ambient auth, no MCP, and no task subagents
  • block system/MDM-managed OpenCode config and reject pack-declared OpenCode/XDG control variables before binary detection
  • delete the binary session database after recovery
  • omit tool outputs, errors, and reasoning text from durable JSONL; exact-redact every harness-visible credential and pack value
  • fail closed on malformed result shapes, missing trace evidence, unsafe short values, timeouts, and harness mismatch
  • record a bounded controller-derived outcome reason for every retry attempt
  • preserve --task in the generated verify command so single-task pilots use the correct denominator
  • reject objectively observed vendor-CLI actions in an API cell as cross-surface evidence

Live pilots

  • OpenCode 1.18.3 with opencode/deepseek-v4-flash-free
  • direct smoke emitted native JSONL and wrote the exact requested file; it caught that 1.18.3 rejects --no-env-file, fixed in e7d5ac3
  • Exa example-pack API task made a real Search call and passed live Contents read-back: 1/1, 100%; updated normalization reports 7 tool calls across 28 events
  • InsForge DAEB research task created and inspected the exact run-scoped table; arena SQL read-back found both expected fields and bounded reset deleted 1/1 resource
  • the DAEB transcript also used npx @insforge/cli in an API cell; updated verification now reports this as cross-surface FAIL instead of a clean API pass, and reports 34 tool calls across 103 events
  • DAEB remains research evidence only: this generic integration does not add OpenCode to the canonical arena production policy or publication matrix

Validation

  • core Vitest: 58 files, 540 tests passed
  • arena Vitest: 47 files, 285 tests passed with one worker (default macOS parallel run hit process-launch timeouts; Linux CI is authoritative)
  • npm run typecheck
  • npm run build
  • npm --cache .npm-cache pack --dry-run
  • real Exa single-task live verification: 100% (1/1), gate passed
  • DAEB transcript replay: correct 1-task denominator, tool count, and cross-surface failure

Stacked on #217; merge that PR first.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

Normalized records diff

✅ PASS — No correctness regression detected

Compared 4 baseline surface cell(s); no existing pass@1 or pass³ score decreased or disappeared.

Overall is a macro-average of participating surface scores. Agents are never averaged together.

Overall by agent

Product Agent Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Agent Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect the verdict or ranking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant