feat(evals): inspect-ai harness + promptfoo merge gate + bench-off scaffold#1
Merged
Conversation
…affold Workstream C (research-implementation program 2026-07-08): - inspect-ai==0.3.244 (PR #11 anchor) uv project; benchoff task (6-sample plumbing-scale set — grow before treating deltas as adoption evidence); .eval transcripts canonical, export_summary.py renders public/data JSON. Live-verified through the gateway (2 samples, accuracy 1.0). Gotcha documented in-code: use openai-api/llm/... — the plain openai provider speaks the Responses API, which upstream lanes 404. - promptfoo eval-gate workflow (SHA-pinned action + checkout): threshold 80, repeat 2, pass-1-of-2, disk cache; scoped to evals/** PRs; self-skips with a warning until the UNSIGNED_LLM_BENCH_KEY secret exists (deploy-token gate pattern). - Prereqs documented: paas #880 mints the capped bench key + MiniMax lane.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workstream C (final wave-1 piece). Depends on paas#880 for the capped bench key + minimax lane; until the
UNSIGNED_LLM_BENCH_KEYsecret is added the gate self-skips with a warning.Verified: uv sync clean at the 0.3.244 anchor; task builds (6 samples); live 2-sample smoke through llm.unsigned.gg → accuracy 1.0; exporter renders dashboard JSON from the real .eval; both action pins verified against GitHub (my first from-memory checkout SHA was wrong — caught and fixed with the real v7.0.0 pin).
Operator steps: add
UNSIGNED_LLM_BENCH_KEYsecret (after #880 sync) ·CLOUDFLARE_API_TOKENstill pending separately.Merge = operator.