Add OSWorld-V2 hybrid GUI+CLI evaluation experiment

README.md

@@ -14,6 +14,7 @@
 
 ## 📰 News
 
+- **2026-06-30** — **OSWorld-V2** under a **hybrid GUI+CLI harness** (codex CLI in-VM + GUI, native grading): GPT-5.5 **+5.5 pt Binary (13.0→18.5%)** at **~2× tool-call efficiency (149.8→77.5/task)**. [`experiments/osworld_v2_hybrid/`](./experiments/osworld_v2_hybrid).
 - **2026-06-12** — WeaveBench hit **#4 on [Hugging Face Daily Papers](https://huggingface.co/papers/2606.09426)** (104 upvotes). 🎉
 - **2026-06-10** — Evaluated **9 frontier backbones × 4 agent runtimes** (OpenClaw, Codex CLI, Claude Code, Hermes); best pairing tops out at **41.2% PassRate**. Full leaderboard in [`docs/REPRODUCE.md`](./docs/REPRODUCE.md).
 - **2026-06-08** — Initial preprint and [project website](https://weavebench.github.io) live.

experiments/osworld_v2_hybrid/.gitignore

@@ -0,0 +1,22 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.ipynb_checkpoints/
+
+# Large runtime assets — never commit (download / build separately, see README)
+*.tar.gz
+*.qcow2
+*.zip
+
+# Raw run outputs — keep only the aggregated files under results/codex_hybrid_gpt55/
+results/osworld_v2_inject/
+**/pyautogui/
+**/screenshots/
+**/agent.log
+*.log
+
+# Secrets / local endpoints
+*.env
+my_api.json
+litellm_api.json

experiments/osworld_v2_hybrid/README.md

@@ -0,0 +1,45 @@
+# OSWorld-V2 hybrid GUI+CLI evaluation
+
+A harness ablation on **OSWorld-V2** (108 tasks): we drive each task with a
+**hybrid GUI+CLI agent** — the OpenAI `codex` CLI injected inside the VM plus a
+GUI action channel — on the same **GPT-5.5** backbone the paper benchmarks.
+Scored with OSWorld-V2's own evaluator (`env.evaluate()`, checkpoint-based with
+bounded model judgment) — the same grader as the paper's Table 3, so the numbers
+are directly comparable to the official GPT-5.5 row.
+
+## Headline
+
+| Model / harness | Binary (%) | Partial (%) | Tool calls/task |
+|---|---|---|---|
+| **GPT-5.5 + codex hybrid (this work)** | **18.5** | 49.6 | 77.5 |
+| GPT-5.5 batched (official Table 3) | 13.0 | 49.5 | 149.8 |
+
+Same backbone, swapping the official batched loop for the codex hybrid harness
+lifts GPT-5.5 **13.0% → 18.5% Binary** at ~half the tool calls, with partial
+credit holding steady (49.5 → 49.6) — the gains come from pushing near-complete
+tasks over the line, not from broad partial progress. Dropping 4
+infra-failure tasks (063/064/082/069) gives a 104-task cohort: **51.39% avg /
+19.23% Binary**. Full breakdown in
+[`results/codex_hybrid_gpt55/RESULT_ANALYSIS.md`](./results/codex_hybrid_gpt55/RESULT_ANALYSIS.md).
+
+## Layout
+
+```
+run_osworld_v2_inject.py   # entrypoint: run agent on OSWorld-V2, score via native env.evaluate()
+aggregate_results.py       # rebuild results/*.json from per-task score.json
+mm_agents/codex_agent.py   # the codex CLI + GUI hybrid agent (uses openclaw_agent's VM helpers)
+launchers/                 # exact run commands (env-var placeholders for secrets/paths)
+results/codex_hybrid_gpt55/  # aggregated scores + analysis (raw trajectories not committed)
+```
+
+## Running
+
+Drop this folder into an OSWorld-V2 checkout (needs `desktop_env`, the VM qcow2,
+and an OpenAI-compatible endpoint), then set the launcher placeholders via env
+vars and run:
+
+```bash
+OSWORLD_QCOW2=/path/to/osworld-v2.qcow2 WEAVEBENCH_ASSETS_DIR=/path/to/runtime_assets \
+LITELLM_API_KEY=<your-key> CODEX_REASONING_EFFORT=xhigh \
+bash launchers/run_osworld_v2_inject_codex.sh
+```

experiments/osworld_v2_hybrid/aggregate_results.py

@@ -0,0 +1,160 @@
+#!/usr/bin/env python3
+"""Aggregate OSWorld-V2 codex-hybrid (GPT-5.5) per-task scores into
+per_task_scores.json + summary.json.
+
+Source of truth: each task's score.json written by the inject runner under
+  <run_dir>/pyautogui/screenshot/<model>/tasks/<task_id>/score.json
+plus the action-mix CSV (CLI vs GUI tool calls) and codex total-token counts
+parsed from each task's agent.log.
+
+Run from this folder:
+  RUN_DIR=/path/to/FULL_108_gpt55_codex_xhigh_20260628_125852 \
+  python aggregate_results.py
+
+If RUN_DIR is unset it falls back to the path used for the published numbers.
+The committed JSON in results/codex_hybrid_gpt55/ was produced by this script;
+re-running it against the same run reproduces those files exactly.
+"""
+import csv
+import json
+import os
+import re
+import statistics as st
+from pathlib import Path
+
+RUN_DIR = Path(
+    os.environ.get(
+        "RUN_DIR",
+        # Point this at your inject-runner output directory, e.g.
+        #   results/osworld_v2_inject/OSW2_codex_hybrid_<timestamp>
+        "results/osworld_v2_inject/FULL_108_gpt55_codex_xhigh_20260628_125852",
+    )
+)
+MODEL = os.environ.get("MODEL", "gpt-5.5")
+OUT_DIR = Path(__file__).parent / "results" / "codex_hybrid_gpt55"
+
+# Infra-failure tasks excluded from the de-infra cohort (root cause unrelated to
+# model capability): VM disk fault, network resets, framework multiphase limit.
+INFRA = {"063", "064", "082", "069"}
+INFRA_REASON = {
+    "063": "VM ext4 journal error -> root remounted read-only",
+    "064": "network: Connection broken / IncompleteRead during execution",
+    "082": "network: Connection reset by peer during task setup",
+    "069": "framework: multiphase_unsupported_in_inject_runner",
+}
+
+TOK_RE = re.compile(r"^([0-9,]+)$")
+
+
+def parse_tokens(agent_log: Path):
+    """Sum codex 'tokens used' totals across turns (total tok = input+reasoning+output)."""
+    if not agent_log.exists():
+        return None
+    toks = []
+    lines = agent_log.read_text(errors="replace").splitlines()
+    for i, l in enumerate(lines):
+        if l.strip() == "tokens used" and i + 1 < len(lines):
+            m = TOK_RE.match(lines[i + 1].strip())
+            if m:
+                toks.append(int(m.group(1).replace(",", "")))
+    return sum(toks) if toks else None
+
+
+def load_action_mix():
+    csv_path = OUT_DIR / "action_mix.csv"
+    if not csv_path.exists():
+        return {}
+    mix = {}
+    with open(csv_path) as f:
+        for row in csv.DictReader(f):
+            mix[row["task_id"]] = {
+                "cli": int(row["cli"]),
+                "gui": int(row["gui"]),
+                "total": int(row["total"]),
+            }
+    return mix
+
+
+def main():
+    tasks_dir = RUN_DIR / "pyautogui" / "screenshot" / MODEL / "tasks"
+    mix = load_action_mix()
+    rows = []
+    for td in sorted(tasks_dir.glob("*")):
+        sj = td / "score.json"
+        if not sj.exists():
+            continue
+        s = json.loads(sj.read_text())
+        tid = s["task_id"]
+        rows.append(
+            {
+                "task_id": tid,
+                "score": s.get("score") or 0.0,
+                "agent_done": s.get("agent_done"),
+                "error": s.get("error"),
+                "elapsed_seconds": s.get("elapsed_seconds"),
+                "total_tokens": parse_tokens(td / "agent.log"),
+                "tool_calls": mix.get(tid, {}).get("total"),
+                "gui_calls": mix.get(tid, {}).get("gui"),
+                "cli_calls": mix.get(tid, {}).get("cli"),
+            }
+        )
+
+    def cohort(keep):
+        n = len(keep)
+        binary = sum(1 for r in keep if r["score"] >= 0.999)
+        avg = sum(r["score"] for r in keep) / n
+        nonzero = sum(1 for r in keep if 0 < r["score"] < 0.999)
+        return {
+            "n": n,
+            # Paper Table 3 metrics: Binary = fraction at score 1.0; Partial =
+            # mean partial-credit score over all tasks (NOT the fraction of
+            # tasks with a partial score).
+            "binary_pct": round(binary / n * 100, 2),
+            "partial_pct": round(avg * 100, 2),
+            "binary_count": binary,
+            "nonzero_partial_count": nonzero,
+        }
+
+    def mean(key, src=rows):
+        vs = [r[key] for r in src if r.get(key) is not None]
+        return round(sum(vs) / len(vs), 1) if vs else None
+
+    def median(key, src=rows):
+        vs = [r[key] for r in src if r.get(key) is not None]
+        return round(st.median(vs), 1) if vs else None
+
+    all108 = cohort(rows)
+    deinfra = cohort([r for r in rows if r["task_id"] not in INFRA])
+
+    summary = {
+        "run": RUN_DIR.name,
+        "model": MODEL,
+        "harness": "codex CLI injected into VM + GUI channel (hybrid)",
+        "reasoning_effort": "xhigh",
+        "scoring": "native OSWorld env.evaluate() (paper-faithful)",
+        "cohorts": {"all_108": all108, "de_infra_104": deinfra},
+        "efficiency": {
+            "tool_calls_per_task": {"mean": mean("tool_calls"), "median": median("tool_calls")},
+            "gui_calls_per_task": {"mean": mean("gui_calls"), "median": median("gui_calls")},
+            "cli_calls_per_task": {"mean": mean("cli_calls"), "median": median("cli_calls")},
+            "total_tokens_per_task_incl_input": {
+                "mean": mean("total_tokens"),
+                "median": median("total_tokens"),
+                "note": "codex CLI logs only a combined token total (input+reasoning+output); "
+                "output-only tokens and cost were not recorded -> reported as '-' in tables",
+            },
+        },
+        "excluded_infra_tasks": INFRA_REASON,
+    }
+
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUT_DIR / "per_task_scores.json").write_text(json.dumps(rows, indent=2, ensure_ascii=False))
+    (OUT_DIR / "summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False))
+    print("wrote", OUT_DIR / "per_task_scores.json", "(", len(rows), "tasks )")
+    print("wrote", OUT_DIR / "summary.json")
+    print("all_108:", all108)
+    print("de_infra_104:", deinfra)
+
+
+if __name__ == "__main__":
+    main()

experiments/osworld_v2_hybrid/launchers/run_osworld_v2_inject_claude.sh

@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+# OSWorld-V2 (108 tasks) — CLAUDE CODE injected, CLI-ONLY agent.
+# In-VM agent = `claude` CLI + opus-4.8 at MAX thinking (paper Table 3 parity).
+# CLI-only (AGENT_GUI=false): no computer MCP, pure shell/file tools.
+set -euo pipefail
+
+LAUNCHER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EVAL_DIR="$(cd "${LAUNCHER_DIR}/.." && pwd)"
+OSWORLD_DIR="$(cd "${EVAL_DIR}/../.." && pwd)"
+
+# === Anthropic endpoint (copilot-api on 4141) ===
+LITELLM_HOST_IP="${LITELLM_HOST_IP:-127.0.0.1}"
+LITELLM_VM_IP="${LITELLM_VM_IP:-172.17.0.1}"
+LITELLM_PORT="${LITELLM_PORT:-4141}"
+LITELLM_API_KEY="${LITELLM_API_KEY:-dummy}"
+MODEL="${MODEL:-claude-opus-4-8}"
+
+# === Run knobs ===
+NUM_ENVS="${NUM_ENVS:-8}"
+MAX_STEPS="${MAX_STEPS:-500}"
+AGENT_GUI="${AGENT_GUI:-false}"          # CLI-only
+AGENT_TIMEOUT="${AGENT_TIMEOUT:-7200}"
+LIMIT="${LIMIT:-0}"
+TASK_FILTER="${TASK_FILTER:-}"
+CLIENT_PASSWORD="${CLIENT_PASSWORD:-osworld-public-evaluation}"
+
+# === claude-specific ===
+export CLAUDE_EFFORT_LEVEL="${CLAUDE_EFFORT_LEVEL:-max}"
+export CLAUDE_MAX_RETRIES="${CLAUDE_MAX_RETRIES:-3}"
+
+# === Image / data ===
+OSWORLD_QCOW2="${OSWORLD_QCOW2:-/path/to/osworld_v2_images/osworld-v2-ubuntu-x86.qcow2}"
+export WEAVEBENCH_ASSETS_DIR="${WEAVEBENCH_ASSETS_DIR:-/path/to/osworld_v2_images/runtime_assets}"
+export WEBSITE_HOST_SUFFIX="${WEBSITE_HOST_SUFFIX:-web.hku.icu}"
+
+# host-side LLM-judge evaluators still route via LiteLLM 4200 (gpt judge)
+export OSWORLD_EVAL_MODEL_PROVIDER="${OSWORLD_EVAL_MODEL_PROVIDER:-openai_compatible}"
+export OSWORLD_EVAL_MODEL_API_KEY="${OSWORLD_EVAL_MODEL_API_KEY:-sk-litellm-azure-direct}"
+export OSWORLD_EVAL_MODEL_BASE_URL="${OSWORLD_EVAL_MODEL_BASE_URL:-http://127.0.0.1:4200/v1}"
+
+export GITLAB_URL="${GITLAB_URL:-http://172.17.0.1}"
+export GITLAB_PRIVATE_TOKEN="${GITLAB_PRIVATE_TOKEN:-REPLACE_WITH_YOUR_GITLAB_TOKEN}"
+
+RESULT_TAG="${RESULT_TAG:-OSW2_claude_cli_$(date +%Y%m%d_%H%M%S)}"
+RESULT_DIR="${RESULT_DIR:-${OSWORLD_DIR}/results/osworld_v2_inject/${RESULT_TAG}}"
+mkdir -p "${RESULT_DIR}"
+PY_BIN="${PY_BIN:-python3}"
+
+# === Preflight ===
+echo "[preflight] copilot-api ${LITELLM_HOST_IP}:${LITELLM_PORT} has ${MODEL}?"
+curl -sS --max-time 8 --noproxy '*' "http://${LITELLM_HOST_IP}:${LITELLM_PORT}/v1/models" \
+  | grep -q "${MODEL}" || { echo "[preflight] FAIL: ${MODEL} not on 4141"; exit 1; }
+echo "[preflight] opus OK"
+[ -f "${OSWORLD_QCOW2}" ] || { echo "[preflight] FAIL: qcow2 missing"; exit 1; }
+CCTAR="${WEAVEBENCH_ASSETS_DIR}/claudecode.tar.gz"
+[ -f "$CCTAR" ] || CCTAR="/path/to/runtime_assets/claudecode.tar.gz"
+[ -f "$CCTAR" ] || { echo "[preflight] FAIL: claudecode.tar.gz missing"; exit 1; }
+echo "[preflight] claudecode assets OK ($CCTAR)"
+[ -d "${OSWORLD_DIR}/evaluation_examples/task_class" ] || { echo "[preflight] FAIL task_class"; exit 1; }
+echo "[preflight] task_class OK"
+
+# in-VM reaches copilot-api on host gateway:4141, no /v1 suffix (anthropic SDK adds it)
+AGENT_BASE_URL="http://${LITELLM_VM_IP}:${LITELLM_PORT}"
+
+echo "==========================================================="
+echo "  OSWorld-V2 CLAUDE CODE (CLI-only) — ${MODEL} effort=${CLAUDE_EFFORT_LEVEL}"
+echo "Agent: ${AGENT_BASE_URL}  gui=${AGENT_GUI}  envs=${NUM_ENVS}  timeout=${AGENT_TIMEOUT}s"
+echo "Result: ${RESULT_DIR}  Start: $(date)"
+echo "==========================================================="
+
+cd "${OSWORLD_DIR}"
+HOST_PROXY="${HOST_PROXY:-http://127.0.0.1:7897}"
+export http_proxy="${HOST_PROXY}" https_proxy="${HOST_PROXY}"
+export HTTP_PROXY="${HOST_PROXY}" HTTPS_PROXY="${HOST_PROXY}"
+export NO_PROXY="127.0.0.1,localhost,172.17.0.1,0.0.0.0"; export no_proxy="${NO_PROXY}"
+
+EXTRA_ARGS=()
+[ -n "${TASK_FILTER}" ] && EXTRA_ARGS+=(--task_filter "${TASK_FILTER}")
+[ "${LIMIT}" -gt 0 ] 2>/dev/null && EXTRA_ARGS+=(--limit "${LIMIT}")
+[ -n "${TEST_META_PATH:-}" ] && EXTRA_ARGS+=(--test_all_meta_path "${TEST_META_PATH}")
+
+"${PY_BIN}" "${EVAL_DIR}/run_osworld_v2_inject.py" \
+  --provider_name docker --headless true \
+  --path_to_vm "${OSWORLD_QCOW2}" --osworld_root "${OSWORLD_DIR}" \
+  --num_envs "${NUM_ENVS}" --model "${MODEL}" \
+  --litellm_base_url "${AGENT_BASE_URL}" --litellm_api_key "${LITELLM_API_KEY}" \
+  --max_steps "${MAX_STEPS}" --agent_gui "${AGENT_GUI}" \
+  --agent_timeout "${AGENT_TIMEOUT}" --client_password "${CLIENT_PASSWORD}" \
+  --result_dir "${RESULT_DIR}" --agent_harness claudecode \
+  "${EXTRA_ARGS[@]}" \
+  2>&1 | tee "${RESULT_DIR}/run_$(date +%Y%m%d_%H%M%S).log"