feat(evals): inspect-ai harness + promptfoo merge gate + bench-off scaffold

.github/workflows/eval-gate.yml

@@ -0,0 +1,46 @@
+# Merge gate for eval-definition changes (gap-closure verdict 2026-07-07:
+# promptfoo Action = the OSS merge gate; Inspect AI stays the deep-eval layer).
+# Scoped to evals/** so dashboard PRs never burn tokens. Uses the capped bench
+# key (OPS-465 unattended lane) — NEVER the shared interactive key.
+name: eval-gate
+
+on:
+  pull_request:
+    paths:
+      - "evals/**"
+
+permissions:
+  contents: read
+  pull-requests: write # PR summary comment
+
+jobs:
+  promptfoo:
+    runs-on: ubuntu-latest
+    # Skip cleanly until the operator adds the bench key secret (mirrors the
+    # deploy workflow's token gate pattern).
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - name: bench key present?
+        id: key
+        run: |
+          if [ -n "${{ secrets.UNSIGNED_LLM_BENCH_KEY }}" ]; then
+            echo "present=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "::warning::UNSIGNED_LLM_BENCH_KEY secret not set — eval gate skipped"
+            echo "present=false" >> "$GITHUB_OUTPUT"
+          fi
+      - name: promptfoo eval
+        if: steps.key.outputs.present == 'true'
+        uses: promptfoo/promptfoo-action@04839e664f52212b877170219dab0d7ad2bf6440 # v1.3
+        with:
+          config: evals/promptfooconfig.yaml
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          # Flakiness handling per the verified pattern: pass-k-of-n tolerates
+          # grader/model variance; consistent failures still gate.
+          fail-on-threshold: "80"
+          repeat: "2"
+          repeat-min-pass: "1"
+          # Default disk cache; prune stale entries in CI.
+          cache-path: ~/.promptfoo/cache
+        env:
+          UNSIGNED_LLM_BENCH_KEY: ${{ secrets.UNSIGNED_LLM_BENCH_KEY }}

.gitignore

@@ -1,2 +1,6 @@
 .wrangler/
 node_modules/
+
+# evals
+evals/logs/
+evals/.venv/

evals/README.md

@@ -0,0 +1,44 @@
+<!-- lineage
+role: eval-harness-readme
+conforms_to: ../README.md
+consumes: agentic docs/eval/2026-07-07-* (verdicts: inspect-ai anchor, promptfoo gate, bench-off rule)
+-->
+
+# evals — inspect-ai harness + promptfoo merge gate
+
+Greenfield lane from the 2026-07-08 research-implementation program. Two
+layers, per the gap-closure verdicts:
+
+- **inspect-ai** (`inspect-ai==0.3.244`, the PR #11 anchor) — deep evals.
+  `.eval` transcripts under `logs/` are the **canonical evidence**;
+  `export_summary.py` renders them into `public/data/benchoff.json` for the
+  dashboard. First task: `tasks/benchoff.py` (MiniMax-M2.7 vs GLM-5.2 vs
+  Kimi-K2.7-Code — a plumbing-scale task-set; grow it before treating deltas
+  as adoption evidence).
+- **promptfoo** (`.github/workflows/eval-gate.yml`, SHA-pinned action) —
+  merge-blocking gateway regression on `evals/**` PRs: threshold 80, repeat 2,
+  pass-1-of-2 flake tolerance, disk cache. Self-skips with a warning until the
+  `UNSIGNED_LLM_BENCH_KEY` secret exists.
+
+## Run the bench-off
+
+```bash
+cd evals && uv sync
+export LLM_BASE_URL=https://llm.unsigned.gg/v1
+export LLM_API_KEY=$UNSIGNED_LLM_BENCH_KEY   # capped OPS-465 lane key
+# openai-api/<service>/... pins chat-completions; the plain openai provider
+# uses the Responses API which upstream lanes 404 (verified 2026-07-08).
+uv run inspect eval tasks/benchoff.py \
+  --model openai-api/llm/zai/GLM-5.2 \
+  --model openai-api/llm/moonshotai/kimi-k2.7-code \
+  --model openai-api/llm/minimax/minimax-m2.7
+uv run python export_summary.py
+```
+
+## Prereqs (operator)
+
+1. paas PR #880 synced (mints the capped `bench` virtual key + adds the
+   `minimax/minimax-m2.7` lane).
+2. Repo secret `UNSIGNED_LLM_BENCH_KEY` = the bench key value (from OpenBao
+   `secret/llm` property `bench-api-key`).
+3. `CLOUDFLARE_API_TOKEN` still pending separately for the deploy job.

evals/export_summary.py

@@ -0,0 +1,52 @@
+"""Export inspect .eval transcripts into the dashboard's data dir.
+
+Reads every log under evals/logs/, aggregates per-model accuracy + sample
+count + token usage, writes public/data/benchoff.json for the existing static
+dashboard. The .eval files remain the canonical evidence; this JSON is a view.
+
+    uv run python export_summary.py [--logs logs] [--out ../public/data/benchoff.json]
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+from inspect_ai.log import list_eval_logs, read_eval_log
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--logs", default="logs")
+    ap.add_argument("--out", default="../public/data/benchoff.json")
+    args = ap.parse_args()
+
+    results = {}
+    for info in list_eval_logs(args.logs):
+        log = read_eval_log(info, header_only=True)
+        if log.status != "success" or not log.results:
+            continue
+        model = log.eval.model
+        accuracy = None
+        for score in log.results.scores:
+            for name, metric in score.metrics.items():
+                if name == "accuracy":
+                    accuracy = metric.value
+        usage = log.stats.model_usage.get(model)
+        results[model] = {
+            "task": log.eval.task,
+            "accuracy": accuracy,
+            "samples": log.results.total_samples,
+            "input_tokens": usage.input_tokens if usage else None,
+            "output_tokens": usage.output_tokens if usage else None,
+            "run_id": log.eval.run_id,
+            "created": log.eval.created,
+        }
+
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps({"benchoff": results}, indent=2, sort_keys=True) + "\n")
+    print(f"wrote {out} ({len(results)} models)")
+
+
+if __name__ == "__main__":
+    main()

evals/promptfooconfig.yaml

@@ -0,0 +1,35 @@
+# Gateway regression gate — the merge-blocking promptfoo config.
+# Runs on eval-def PRs only (see .github/workflows/eval-gate.yml). Tiny by
+# design: a handful of deterministic asserts against two gateway lanes proves
+# routing + auth + basic model sanity without burning tokens on every PR.
+# Anthropic lanes deliberately excluded (cache-sticky path stays clean).
+
+providers:
+  - id: openai:chat:zai/GLM-5.2
+    config:
+      apiBaseUrl: https://llm.unsigned.gg/v1
+      apiKeyEnvar: UNSIGNED_LLM_BENCH_KEY
+  - id: openai:chat:moonshotai/kimi-k2.7-code
+    config:
+      apiBaseUrl: https://llm.unsigned.gg/v1
+      apiKeyEnvar: UNSIGNED_LLM_BENCH_KEY
+
+prompts:
+  - "Answer with only the requested value, no prose. {{question}}"
+
+tests:
+  - vars:
+      question: "What is 17 * 3?"
+    assert:
+      - type: contains
+        value: "51"
+  - vars:
+      question: "Name the HTTP status code for Not Found."
+    assert:
+      - type: contains
+        value: "404"
+  - vars:
+      question: "Reply with exactly the word: ok"
+    assert:
+      - type: icontains
+        value: "ok"

evals/pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "unsigned-bench-evals"
+version = "0.1.0"
+description = "inspect-ai eval harness for the unsigned model gateway (bench-off + regression tasks)"
+requires-python = ">=3.11"
+dependencies = [
+    # The PR #11 bench anchor — bump deliberately, PR per bump.
+    "inspect-ai==0.3.244",
+    "openai>=1.60",
+]
+
+[tool.uv]
+package = false

evals/tasks/benchoff.py

@@ -0,0 +1,70 @@
+"""Bench-off task: MiniMax-M2.7 vs GLM-5.2 vs Kimi-K2.7-Code via the gateway.
+
+Scaffold task-set (wave-1, 2026-07-08 program): small, deterministic,
+string-verifiable coding samples — enough to exercise the full lane
+(gateway -> model -> scorer -> .eval transcript -> dashboard JSON), NOT a
+definitive capability ranking. Grow the dataset before treating deltas as
+adoption evidence (the survey's rule: verify vendor numbers, never adopt on
+marketing claims).
+
+Run (bench key = the capped OPS-465 lane, never the shared interactive key):
+
+    export LLM_BASE_URL=https://llm.unsigned.gg/v1
+    export LLM_API_KEY=$UNSIGNED_LLM_BENCH_KEY
+    uv run inspect eval tasks/benchoff.py \
+        --model openai-api/llm/zai/GLM-5.2 \
+        --model openai-api/llm/moonshotai/kimi-k2.7-code \
+        --model openai-api/llm/minimax/minimax-m2.7
+    # NOTE: openai-api/<service>/ (not openai/) — the plain openai provider
+    # speaks the Responses API, which the gateway's upstream lanes 404
+    # (verified live 2026-07-08); openai-api pins chat-completions.
+
+.eval transcripts under logs/ are the canonical evidence (model-infra gap
+survey decision); export_summary.py turns them into public/data/benchoff.json.
+"""
+
+from inspect_ai import Task, task
+from inspect_ai.dataset import Sample
+from inspect_ai.scorer import includes
+from inspect_ai.solver import generate, system_message
+
+SYSTEM = (
+    "You are a precise coding assistant. Answer with ONLY the requested "
+    "output — no prose, no markdown fences."
+)
+
+SAMPLES = [
+    Sample(
+        input="Write a Python one-liner expression (no assignment) that reverses the string s. Reply with only the expression.",
+        target="s[::-1]",
+    ),
+    Sample(
+        input="What does this Python print? `print(sorted({'b': 2, 'a': 1}))` Reply with only the printed output.",
+        target="['a', 'b']",
+    ),
+    Sample(
+        input="Give the exact jq filter to extract the field `id` from every element of a top-level JSON array. Reply with only the filter.",
+        target=".[].id",
+    ),
+    Sample(
+        input="In git, what single flag makes `git log` show one commit per line? Reply with only the flag.",
+        target="--oneline",
+    ),
+    Sample(
+        input="What is the exit status of `bash -c 'false || true'`? Reply with only the number.",
+        target="0",
+    ),
+    Sample(
+        input="Complete: a Kubernetes CronJob field that prevents overlapping runs is `concurrencyPolicy: ___`. Reply with only the value.",
+        target="Forbid",
+    ),
+]
+
+
+@task
+def benchoff() -> Task:
+    return Task(
+        dataset=SAMPLES,
+        solver=[system_message(SYSTEM), generate()],
+        scorer=includes(),
+    )