Skip to content

Add design/screenshot probe type (render + visual judge)#12

Merged
kwadwoadu merged 1 commit into
mainfrom
feat/design-screenshot-probes
Jul 15, 2026
Merged

Add design/screenshot probe type (render + visual judge)#12
kwadwoadu merged 1 commit into
mainfrom
feat/design-screenshot-probes

Conversation

@kwadwoadu

Copy link
Copy Markdown
Owner

What

Adds a first-class scoring: screenshot probe type so modelfit can benchmark models on UI/design work. The candidate model returns a self-contained HTML artifact, modelfit renders it headlessly to a PNG, and a multimodal judge grades the rendered image against the rubric. Text probes are unchanged (fully backward compatible).

Flow

design probe -> model returns HTML -> render (Playwright CLI / Chromium) -> result.png
                                                                                |
                verdict <- multimodal judge (sees the image) <-----------------+

Changes

  • common.sh: scoring_for_probe, b64_file, render_html (MODELFIT_RENDER_CMD override -> Playwright CLI -> Chromium/Chrome, falling through on failure; MODELFIT_RENDER_VIEWPORT)
  • run.sh: screenshot probes write result.html, render to result.png, record rendered:true; render failure -> render_error
  • judge.sh: multimodal judge (anthropic image / openai image_url) via jq --rawfile; body streamed over stdin (--data @-) to avoid the 128KB single-arg limit on base64 images; text-only fallback when no PNG
  • bin/render.sh + modelfit render dispatch
  • probes/example-design.md, prompt/README/limitations docs
  • tests/design.test.sh (mock renderer + provider, zero browser/network); mock tests/curl reads --data @-; selftest covers the new paths

Verification

  • ./bin/selftest.sh PASS, shellcheck CLEAN
  • End-to-end proven live: a model produced pricing-card HTML -> rendered -> multimodal judge graded the pixels (pass, quality 5)

Probes marked 'scoring: screenshot' now render the candidate model's HTML
output headlessly to a PNG and grade the rendered image with a multimodal
judge, instead of grading source text. Text probes are unchanged.

- common.sh: scoring_for_probe, b64_file, render_html (MODELFIT_RENDER_CMD
  override -> Playwright CLI -> Chromium/Chrome, falling through on failure;
  MODELFIT_RENDER_VIEWPORT)
- run.sh: screenshot probes write result.html, render to result.png, record
  rendered:true; render failure -> render_error (honest coverage gap)
- judge.sh: multimodal judge (anthropic image / openai image_url) built via
  jq --rawfile; body sent over stdin (--data @-) to dodge the 128KB single-arg
  limit on base64 images; falls back to text-only when no PNG
- bin/render.sh + 'modelfit render' dispatch
- probes/example-design.md, prompts + README + limitations docs
- tests/design.test.sh (mock renderer + provider, zero browser/network);
  tests/curl mock reads --data @-; selftest covers the new paths
@kwadwoadu
kwadwoadu merged commit de9cfd2 into main Jul 15, 2026
1 check passed
@kwadwoadu
kwadwoadu deleted the feat/design-screenshot-probes branch July 15, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant