Skip to content

Run eval cases in parallel with a concurrency limit #10

Description

@Cryptoteep

Context

evalPrompt currently runs cases sequentially in a for loop:

// src/eval.ts
for (const c of cases) {
  // await runPrompt(...)
  // await c.assert(result)
}

For large eval suites against a real model, this is slow. Most providers handle moderate concurrency fine.

Proposal

Add a concurrency option to evalPrompt (default 1 to preserve current behaviour). When > 1, run cases through a bounded concurrency pool.

const result = await evalPrompt(prompt, cases, {
  call,
  concurrency: 8,
})

Implementation notes

  • Implement a tiny bounded pool (no deps) — p-limit-style but inline.
  • Preserve case ordering in the result (cases[] must match input order, not completion order).
  • Keep durations accurate per case.
  • Consider onCaseStart / onCaseDone callbacks for live reporters (nice-to-have, separate issue).

Acceptance

  • concurrency option added, default 1.
  • Result order matches input order regardless of completion order.
  • Test with a mock that sleeps, asserting wall-clock time drops with higher concurrency.
  • Docs updated in docs/testing.md.

/label "help wanted" "area:eval" "enhancement"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions