Skip to content

fix: reuse cached results for single-experiment runs#153

Closed
benjamincanac wants to merge 3 commits into
vercel-labs:mainfrom
benjamincanac:fix/single-experiment-reuse
Closed

fix: reuse cached results for single-experiment runs#153
benjamincanac wants to merge 3 commits into
vercel-labs:mainfrom
benjamincanac:fix/single-experiment-reuse

Conversation

@benjamincanac

@benjamincanac benjamincanac commented Jun 11, 2026

Copy link
Copy Markdown

Summary

agent-eval <name> (run one experiment) routed to a separate code path from agent-eval / run-all, and the two disagreed on how results were stored and reused. The single path always nested results by model, never wrote a fingerprint into summary.json, and never scanned for reuse. Net effect: results produced by the documented "run one experiment" command were structurally invisible to resume, so they got re-run on every invocation and the next run-all re-ran everything from scratch.

  • Default action now delegates to runAllCommand, so the single-name case is just run-all filtered to one name. It computes fingerprints, scans for reuse, passes fingerprints into runExperiment, and stores single-model results at results/<name>/<ts>/.
  • Removed the now-dead runExperimentCommand and helpers it owned (resolveConfigPath, plus unused imports).
  • Hardened scanReusableResults to recurse through model-nested directories, so legacy results at results/<name>/<model>/<ts>/ (and normal multi-model layouts) are reused without a manual backfill. The fingerprint encodes the model, so cross-model false reuse can't happen.

Repro (before)

agent-eval my-experiment            # runs, saves results
agent-eval run-all my-experiment --dry
# => "25 to run, 0 cached" even though everything just passed

After the fix, the second command reports the just-passed evals as cached.

Behavior notes

  • Single-experiment accepts bare names, globs, and path-style experiments/cc.ts (normalized to the bare experiment name), so existing invocations keep working.
  • Fixtures are loaded per experiment, so the validation config field (e.g. validation: 'none') is honored by the unified path — previously only the single-experiment path respected it.
  • An explicitly named experiment that can't run (config fails to load, missing API key, no valid fixtures) exits non-zero instead of being skipped with exit 0. No-args sweeps still skip unloadable configs, but an experiment with no valid fixtures fails the sweep's exit code.
  • Infra/timeout failures are still re-run (isNonModelFailure behavior unchanged).

Test plan

  • agent-eval <name> then agent-eval run-all <name> --dry reports cached (new cli.test.ts case)
  • scanReusableResults picks up single-segment and provider/model nested layouts, and prefers newest across mixed layouts (new results.test.ts cases)
  • Path-style experiments/cc.ts argument still works (new cli.test.ts case)
  • validation: 'none' evals without EVAL.ts are discovered (new cli.test.ts case)
  • Named experiment with invalid config or missing API key exits 1 (new cli.test.ts cases)
  • npm run lint, npm run build, npm test (221 passed, 14 skipped)
  • changeset added (patch)

`agent-eval <name>` routed to a separate code path that always nested
results by model, never wrote a fingerprint, and never scanned for
reuse — so single-experiment results were invisible to resume and
everything re-ran on the next run-all.

Delegate the default action to runAllCommand so the single-name case
shares identical fingerprinting, storage, and reuse. Remove the now-dead
runExperimentCommand. Also make scanReusableResults recurse through
model-nested directories so legacy results stored at
results/<name>/<model>/<ts>/ are reused without a manual backfill.
@benjamincanac benjamincanac changed the title Fix single-experiment run bypassing fingerprint reuse fix: reuse cached results for single-experiment runs Jun 11, 2026
…yle args

- Load fixtures per experiment so the validation config field
  (validation: 'none') is honored by the unified run path
- Exit non-zero when an explicitly named experiment cannot run
  (config load failure, missing API key); an experiment with no
  valid fixtures fails the exit code in sweeps too
- Normalize path-style arguments (experiments/cc.ts) to bare names
  instead of dropping support
- Document the housekeeping/reuse layout asymmetry and the
  collectTimestampDirs maxDepth rationale
- Surface spawn errors in the CLI test helper; drop a no-op spread
…nt-reuse

# Conflicts:
#	packages/agent-eval/src/cli.test.ts
#	packages/agent-eval/src/cli.ts
@benjamincanac

Copy link
Copy Markdown
Author

Closing as code as diverged and it can be fixed by using run-all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants