Context
evalPrompt currently runs cases sequentially in a for loop:
// src/eval.ts
for (const c of cases) {
// await runPrompt(...)
// await c.assert(result)
}
For large eval suites against a real model, this is slow. Most providers handle moderate concurrency fine.
Proposal
Add a concurrency option to evalPrompt (default 1 to preserve current behaviour). When > 1, run cases through a bounded concurrency pool.
const result = await evalPrompt(prompt, cases, {
call,
concurrency: 8,
})
Implementation notes
- Implement a tiny bounded pool (no deps) —
p-limit-style but inline.
- Preserve case ordering in the result (
cases[] must match input order, not completion order).
- Keep durations accurate per case.
- Consider
onCaseStart / onCaseDone callbacks for live reporters (nice-to-have, separate issue).
Acceptance
/label "help wanted" "area:eval" "enhancement"
Context
evalPromptcurrently runs cases sequentially in aforloop:For large eval suites against a real model, this is slow. Most providers handle moderate concurrency fine.
Proposal
Add a
concurrencyoption toevalPrompt(default1to preserve current behaviour). When> 1, run cases through a bounded concurrency pool.Implementation notes
p-limit-style but inline.cases[]must match input order, not completion order).onCaseStart/onCaseDonecallbacks for live reporters (nice-to-have, separate issue).Acceptance
concurrencyoption added, default1.docs/testing.md./label "help wanted" "area:eval" "enhancement"