Skip to content

Add JSON and HTML reporters for eval results #11

Description

@Cryptoteep

Context

The CLI promptkit eval command prints a plain-text table to stdout. That's fine for local use, but in CI you often want a machine-readable artifact to upload, trend over time, or render in a PR check.

Proposal

Add a --reporter flag to promptkit eval:

promptkit eval ./evals/sentiment.eval.ts --reporter json --out results.json
promptkit eval ./evals/sentiment.eval.ts --reporter html --out results.html

Reporters:

  • text (default, current behaviour)
  • json — full EvalResult serialised, plus a small summary header
  • html — a self-contained static HTML page with a per-case table

Implementation notes

  • Create src/cli/reporters/ with text.ts, json.ts, html.ts.
  • Define a Reporter interface: report(result: EvalResult): string.
  • The HTML reporter should be a single string template — no external CSS/JS deps. Keep it ~100 lines.
  • Add a --out flag that writes to a file instead of stdout (text reporter still goes to stdout by default).

Acceptance

  • --reporter and --out flags wired into the eval command.
  • All three reporters implemented + unit-tested.
  • docs/cli.md updated with examples.
  • CI uploads the JSON artifact as a smoke test (optional).

/label "good first issue" "help wanted" "area:cli" "area:eval" "enhancement"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions