Skip to content

Refactor transcript decoding behind stable events - #217

Open
chenmingtang830 wants to merge 1 commit into
mainfrom
codex/transcript-decoder-seam
Open

Refactor transcript decoding behind stable events#217
chenmingtang830 wants to merge 1 commit into
mainfrom
codex/transcript-decoder-seam

Conversation

@chenmingtang830

Copy link
Copy Markdown
Owner

Summary

  • decode Claude Code and Codex transcripts through a stable harness-event seam
  • select one decoder for the full transcript and carry the expected harness through production paths
  • expose bounded diagnostics and surface decoder mismatches in cell, CLI, and arena records
  • preserve legacy transcript semantics with parity fixtures

Trajectory spike

Tested @letta-ai/trajectory 0.2.0 against three redacted real transcripts from each supported harness. All six failed normalization, so this PR borrows the decode-to-stable-events architecture without adding the dependency.

Validation

  • root Vitest: 57 files, 498 tests passed
  • arena timeout-prone files rerun serially: 7 files, 46 tests passed
  • npm run typecheck
  • npm run build

@github-actions

Copy link
Copy Markdown

Normalized records diff

Overall is a macro-average of participating surface scores. Agents are never averaged together.

Overall by agent

Product Agent Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Surfaces
fixture-db claude-code 92.9% 92.9% +0.0 pp 78.6% (11/14) 78.6% (11/14) 2
fixture-db codex 78.6% 78.6% +0.0 pp 64.3% (9/14) 64.3% (9/14) 2

Surface cells

Product Agent Surface Base pass@1 Head pass@1 Δ Base pass³ Head pass³ Latency ms Cost USD Tokens Harness version
fixture-db claude-code api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1100 → 1100 0.4200 → 0.4200 3350 → 3350 2.1.5 → 2.1.5
fixture-db claude-code cli 100.0% 100.0% +0.0 pp 85.7% (6/7) 85.7% (6/7) 1300 → 1300 0.5100 → 0.5100 3720 → 3720 2.1.5 → 2.1.5
fixture-db codex api 85.7% 85.7% +0.0 pp 71.4% (5/7) 71.4% (5/7) 1200 → 1200 — → — 3600 → 3600 0.121.0 → 0.121.0
fixture-db codex cli 71.4% 71.4% +0.0 pp 57.1% (4/7) 57.1% (4/7) 1500 → 1500 — → — 4100 → 4100 0.121.0 → 0.121.0

Operational metrics are context only and never affect ranking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant