Skip to content

Add normalized-result v2 execution identity - #220

Open
chenmingtang830 wants to merge 1 commit into
codex/opencode-mcpfrom
codex/normalized-result-v2
Open

Add normalized-result v2 execution identity#220
chenmingtang830 wants to merge 1 commit into
codex/opencode-mcpfrom
codex/normalized-result-v2

Conversation

@chenmingtang830

Copy link
Copy Markdown
Owner

Summary:

  • Introduce ax.normalized-result/v2 with product, surface, harness, model, and effort identity while retaining unchanged v1 compatibility.
  • Split producers, records diff, arena selection, and publication exports by model and effort.
  • Reject trial aggregation across execution identity or evaluation-set boundaries.
  • Show model and effort in reviewer-facing diffs and competitive reports.

Migration:

  • New aggregate records emit v2.
  • records-diff continues to read v1; missing effort is shown as unknown.
  • ax.normalized-cell-record/v1 remains unchanged.

Verification:

  • npm run typecheck
  • npm test: core 546, arena 285
  • npm run build
  • npm pack dry run

Stacked on #219.

@github-actions

Copy link
Copy Markdown

Normalized records diff

✅ PASS — No correctness regression detected

Compared 4 baseline surface cell(s); no existing pass@1 or pass³ score decreased or disappeared.

Overall is a macro-average of participating surface scores. Harness/model/effort configurations are never averaged together.

Overall by execution configuration

Product Harness Model Effort Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code claude-sonnet-5 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex gpt-5.6-terra 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Harness Model Effort Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code claude-sonnet-5 api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code claude-sonnet-5 cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex gpt-5.6-terra api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex gpt-5.6-terra cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect the verdict or ranking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant