fix: reuse cached results for single-experiment runs#153
Closed
benjamincanac wants to merge 3 commits into
Closed
Conversation
`agent-eval <name>` routed to a separate code path that always nested results by model, never wrote a fingerprint, and never scanned for reuse — so single-experiment results were invisible to resume and everything re-ran on the next run-all. Delegate the default action to runAllCommand so the single-name case shares identical fingerprinting, storage, and reuse. Remove the now-dead runExperimentCommand. Also make scanReusableResults recurse through model-nested directories so legacy results stored at results/<name>/<model>/<ts>/ are reused without a manual backfill.
…yle args - Load fixtures per experiment so the validation config field (validation: 'none') is honored by the unified run path - Exit non-zero when an explicitly named experiment cannot run (config load failure, missing API key); an experiment with no valid fixtures fails the exit code in sweeps too - Normalize path-style arguments (experiments/cc.ts) to bare names instead of dropping support - Document the housekeeping/reuse layout asymmetry and the collectTimestampDirs maxDepth rationale - Surface spawn errors in the CLI test helper; drop a no-op spread
gaojude
approved these changes
Jun 12, 2026
…nt-reuse # Conflicts: # packages/agent-eval/src/cli.test.ts # packages/agent-eval/src/cli.ts
Author
|
Closing as code as diverged and it can be fixed by using |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent-eval <name>(run one experiment) routed to a separate code path fromagent-eval/run-all, and the two disagreed on how results were stored and reused. The single path always nested results by model, never wrote afingerprintintosummary.json, and never scanned for reuse. Net effect: results produced by the documented "run one experiment" command were structurally invisible to resume, so they got re-run on every invocation and the next run-all re-ran everything from scratch.runAllCommand, so the single-name case is just run-all filtered to one name. It computes fingerprints, scans for reuse, passesfingerprintsintorunExperiment, and stores single-model results atresults/<name>/<ts>/.runExperimentCommandand helpers it owned (resolveConfigPath, plus unused imports).scanReusableResultsto recurse through model-nested directories, so legacy results atresults/<name>/<model>/<ts>/(and normal multi-model layouts) are reused without a manual backfill. The fingerprint encodes the model, so cross-model false reuse can't happen.Repro (before)
After the fix, the second command reports the just-passed evals as cached.
Behavior notes
experiments/cc.ts(normalized to the bare experiment name), so existing invocations keep working.validationconfig field (e.g.validation: 'none') is honored by the unified path — previously only the single-experiment path respected it.isNonModelFailurebehavior unchanged).Test plan
agent-eval <name>thenagent-eval run-all <name> --dryreports cached (newcli.test.tscase)scanReusableResultspicks up single-segment andprovider/modelnested layouts, and prefers newest across mixed layouts (newresults.test.tscases)experiments/cc.tsargument still works (newcli.test.tscase)validation: 'none'evals withoutEVAL.tsare discovered (newcli.test.tscase)cli.test.tscases)npm run lint,npm run build,npm test(221 passed, 14 skipped)