Merge main into the 0.7 line + land OSWorld vendored evaluator

.agents/skills/benchflow-experiment-review/SKILL.md

@@ -31,23 +31,53 @@ Healthy run outcomes are:
 - `pass`: agent completed and verifier produced a valid score.
 - `fail`: agent completed incorrectly and verifier produced a valid score.
 - `normal_timeout`: agent genuinely ran, timed out, and still produced a
-  complete trajectory plus reward/scoring metadata.
+  complete ACP trajectory, complete LLM trajectory, and reward/scoring metadata.
 
 Infrastructure failures are not healthy failures. A stalled Docker daemon,
-Daytona transport failure, missing trajectory, missing reward, missing timing,
-missing token usage for new data, or verifier crash is unhealthy until rerun or
-explicitly quarantined.
+Daytona transport failure, missing `trajectory/acp_trajectory.jsonl`, missing
+`trajectory/llm_trajectory.jsonl`, malformed or empty trajectory files, missing
+reward, missing timing, missing token usage for new data, or verifier crash is
+unhealthy until rerun or explicitly quarantined.
+
+## Hard Trajectory Gate
+
+For every current BenchFlow model trial, both trajectory files are mandatory:
+
+- `trajectory/acp_trajectory.jsonl`: ACP/tool trace with agent-side events.
+- `trajectory/llm_trajectory.jsonl`: provider LLM request/response trace with
+  token usage evidence.
+
+Do not treat ACP alone as sufficient. Do not treat `llm_trajectory.jsonl` alone
+as sufficient. Do not treat aggregate `result.json` fields as a substitute for
+either trajectory. A trial with a scored reward, token counts, or a plausible
+final answer is still unhealthy if either required trajectory file is missing,
+empty, truncated, unparsable, or usage-only without recoverable request/response
+evidence.
+
+Run the bundled validator before deeper manual review:
+
+```bash
+python .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py /path/to/rollout-or-jobs-root --json
+```
+
+The validator exits non-zero when any rollout is unhealthy. It is a deterministic
+fast path, not the whole audit: after it passes, continue with skill-loading,
+no-skill leakage, verifier-isolation, reward-hacking, and capability-attribution
+checks below. Oracle/reward-only lanes do not produce model LLM trajectories and
+must not be mixed into model-run publishability counts; if reviewed, report them
+as separate non-model evidence.
 
 ## Completed Trial Review
 
 For each trial, first locate the authoritative run artifacts. Prefer repo
 validators or existing scripts when available, then inspect raw files. Required
-evidence usually includes:
+evidence includes:
 
 - Run config: task id, harness, model, skill mode, trial id, source commit/ref,
   sandbox backend, timeout, provider settings, and output paths.
-- Trajectories: ACP/tool trajectory and LLM/model trajectory, parseable from
-  first event through final answer, failure, or timeout.
+- Trajectories: both `trajectory/acp_trajectory.jsonl` and
+  `trajectory/llm_trajectory.jsonl`, parseable from first event/model request
+  through final answer, failure, or timeout.
 - Metadata: start/end timestamps, elapsed duration, token usage, tool usage,
   provider/model response metadata, sandbox metadata, and error fields.
 - Results: verifier output, reward/score, result status, and enough provenance
@@ -70,7 +100,8 @@ Review the trajectory for meaning, not just existence:
   without-skill trials should report `0`, meaning task-specific skills were not
   injected into the agent's startup catalog.
 - Timeout trials show real progress or attempts before timeout and still have
-  complete timing, token, trajectory, and verifier result metadata.
+  complete timing, token, ACP trajectory, LLM trajectory, and verifier result
+  metadata.
 - The final answer/result and verifier score refer to the same task workspace
   and trial.
 - Failures and timeouts are attributable to agent capability, not missing
@@ -80,7 +111,11 @@ Review the trajectory for meaning, not just existence:
 
 Reject or quarantine the trial if any of these appear:
 
-- Missing, truncated, or unparsable trajectory files.
+- Missing `trajectory/acp_trajectory.jsonl`.
+- Missing `trajectory/llm_trajectory.jsonl`.
+- Empty, truncated, or unparsable trajectory files.
+- `llm_trajectory.jsonl` has no real provider request bodies, no provider
+  response bodies, or no provider token usage in response metadata.
 - Empty transcript, agent never launched, or only setup logs.
 - Missing token usage/timing/tool usage metadata for newly generated data.
 - The agent lacked required task information, required skills, API keys,
@@ -109,8 +144,9 @@ shape and sandbox behavior.
 3. Run the same canaries through Daytona and VM Docker with the same task,
    harness, model, skill mode, trial id pattern, timeout, and provider settings.
 4. Compare artifact schema and semantics, not exact model wording: trajectory
-   files, token usage, timing, tool usage, result status, verifier output, and
-   source provenance must be equivalent.
+   files (`acp_trajectory.jsonl` and `llm_trajectory.jsonl`), token usage,
+   timing, tool usage, result status, verifier output, and source provenance
+   must be equivalent.
 5. Exercise path/root behavior explicitly. Check task root, sandbox cwd,
    mounted resources, result paths, locked paths, and any previous root-path
    regression case.
@@ -220,6 +256,10 @@ Skill loading mismatch:
 Only export, commit, or push healthy latest trials. Exclude partial runs,
 infra-failed trials, stale duplicates, local caches, and secrets.
 
+Before reporting a trial as healthy or publishable, include the deterministic
+trajectory-gate outcome from `scripts/validate_run_artifacts.py` or equivalent
+manual evidence proving both required trajectory files are present and healthy.
+
 Report findings in this order:
 
 - Blockers: unhealthy trials or integration failures that invalidate data.

.agents/skills/benchflow-experiment-review/evals/evals.json

@@ -72,6 +72,22 @@
         "Output cites missing token usage metadata.",
         "Output cites a missing required API key, resource, dependency, or compute constraint."
       ]
+    },
+    {
+      "id": 5,
+      "prompt": "Use the benchflow-experiment-review skill to audit evals/files/missing-llm-trajectory as a completed Benchflow trial. Return a concise verdict plus the evidence that makes the trial healthy or unhealthy.",
+      "expected_output": "The review rejects the fixture as unhealthy, even though result.json has reward and token_usage, because trajectory/llm_trajectory.jsonl is missing. It cites the hard trajectory contract requiring both ACP and LLM trajectories.",
+      "files": [
+        "evals/files/missing-llm-trajectory/run_config.json",
+        "evals/files/missing-llm-trajectory/result.json",
+        "evals/files/missing-llm-trajectory/trajectory/acp_trajectory.jsonl"
+      ],
+      "expectations": [
+        "Verdict is not publishable, unhealthy, or quarantined.",
+        "Output says missing trajectory/llm_trajectory.jsonl is a hard blocker.",
+        "Output does not treat ACP trajectory alone as sufficient.",
+        "Output cites the hard requirement for both ACP and LLM trajectories."
+      ]
     }
   ]
 }

.agents/skills/benchflow-experiment-review/evals/files/missing-llm-trajectory/result.json

@@ -0,0 +1,17 @@
+{
+  "status": "pass",
+  "reward": 1.0,
+  "verifier_started_after_agent": true,
+  "timing": {
+    "started_at": "2026-06-01T00:00:00Z",
+    "ended_at": "2026-06-01T00:05:00Z",
+    "duration_seconds": 300
+  },
+  "token_usage": {
+    "input_tokens": 1000,
+    "output_tokens": 200
+  },
+  "tool_usage": {
+    "bash": 1
+  }
+}

.agents/skills/benchflow-experiment-review/evals/files/missing-llm-trajectory/run_config.json

@@ -0,0 +1,11 @@
+{
+  "task_id": "example-missing-llm",
+  "harness": "openhands",
+  "model": "sample-model",
+  "skill_mode": "with_skills",
+  "trial_id": 1,
+  "sandbox": "daytona",
+  "source_ref": "eval-fixture",
+  "timeout_seconds": 900,
+  "task_skills": ["data-cleaning"]
+}

.agents/skills/benchflow-experiment-review/evals/files/missing-llm-trajectory/trajectory/acp_trajectory.jsonl

@@ -0,0 +1,3 @@
+{"phase":"agent","type":"tool_call","tool":"bash","args":{"cmd":"python solve.py"}}
+{"phase":"agent","type":"final","content":"Answer submitted."}
+{"phase":"verifier","type":"score","reward":1.0}

.agents/skills/benchflow-experiment-review/references/harness-skill-catalog-sop.md

@@ -1,6 +1,14 @@
 # SOP: recover startup skill catalogs from SkillsBench harness trajectories
 
-The primary source of truth is `trajectory/llm_trajectory.jsonl`; use sibling `trajectory/acp_trajectory.jsonl` as a fallback for the few usage-only LLM logs.
+The primary source of truth for startup skill catalog recovery is
+`trajectory/llm_trajectory.jsonl`; use sibling `trajectory/acp_trajectory.jsonl`
+as a fallback only for the few usage-only LLM logs.
+
+Fallback is not a health waiver. For current BenchFlow model results,
+`trajectory/acp_trajectory.jsonl` and `trajectory/llm_trajectory.jsonl` must both
+exist, be non-empty, parse as JSONL, and contain real evidence. ACP fallback can
+help recover a skill catalog, but it does not make a missing or usage-only
+`llm_trajectory.jsonl` healthy.
 
 The field paths and sample skill counts below are point-in-time observations from the audited dataset. Verify them against the current dataset before treating any count or marker as authoritative.
 
@@ -70,6 +78,14 @@ in nearby metadata such as `run_config.json` / `config.json` keys named
 `required_task_skills`; use repeated `--task-skill <name>` or `--task-path
 <task-dir>` when that metadata is unavailable.
 
+Before skill-catalog extraction, run the hard artifact gate:
+
+```bash
+scripts/validate_run_artifacts.py /path/to/rollout-or-jobs-root --json
+```
+
+If it reports unhealthy, do not mark the trial healthy or publishable.
+
 ## General manual procedure
 
 1. Open `trajectory/llm_trajectory.jsonl`.
@@ -328,19 +344,26 @@ Activation evidence:
 
 PR1/PR2/PR3 v0.1 generally have only `acp_trajectory.jsonl`. For OpenHands, the early ACP `agent_thought` often includes `System Prompt:` and can expose the skill catalog. For Claude Code, Codex, and Gemini CLI v0.1, the startup LLM request is not present, so the full startup system prompt / skill catalog is not reliably recoverable.
 
+Those legacy artifacts can be used as historical aggregate-score references,
+but under the current health contract they are not healthy model trajectories:
+missing `trajectory/llm_trajectory.jsonl` is a hard artifact failure unless the
+row is explicitly scoped as non-model/oracle evidence.
+
 ## Audit checklist
 
 For each trajectory:
 
-1. `llm_trajectory.jsonl` exists and has real request bodies.
-2. Harness detected.
-3. Startup prompt field(s) recorded.
-4. Skill catalog anchor found or explicitly marked absent.
-5. Skill names extracted with count.
-6. `task_skills_loading` checked against run mode: `1` for with-skills, `0`
+1. `acp_trajectory.jsonl` exists, is non-empty, parses, and has agent-side events.
+2. `llm_trajectory.jsonl` exists, is non-empty, parses, has real request bodies,
+   has response bodies, and has provider usage metadata.
+3. Harness detected.
+4. Startup prompt field(s) recorded.
+5. Skill catalog anchor found or explicitly marked absent.
+6. Skill names extracted with count.
+7. `task_skills_loading` checked against run mode: `1` for with-skills, `0`
    for without-skills.
-7. SHA-256 of extracted catalog/startup prompt saved.
-8. For no-skill trials, scan full trajectory for:
+8. SHA-256 of extracted catalog/startup prompt saved.
+9. For no-skill trials, scan full trajectory for:
 
 ```text
 SKILL.md
@@ -352,4 +375,4 @@ activate_skill
 ToolSearch select:
 ```
 
-9. If any of the above appears in a no-skill trajectory, inspect manually before marking healthy.
+10. If any of the above appears in a no-skill trajectory, inspect manually before marking healthy.