From 765e425628c06cb96106470d3b05acdf743465a4 Mon Sep 17 00:00:00 2001 From: Boaz Carmeli Date: Thu, 30 Jul 2026 12:36:18 -0400 Subject: [PATCH] docs(ccc): batch-submission scripts + rootless-podman recipe for CCC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a reproducible userspace path to run cap-evolve on IBM CCC clusters (rootless podman, no admin, no /etc/subuid entry, no systemd on compute nodes). Nine layered workarounds cover image unpack, apt sandboxing, package-postinst chowns/useradds/dpkg-statoverride, private dbus, network_mode=host (aardvark-dns bypass), docker-shim PATH ordering, and docker-compose cp UID mismatches. Ships with a smoke script, a full experiment script (with --resume support for LSF walltime), an LSF submitter, and a colleague-facing README. - docs/RUN_ON_CCC.md — walkthrough + troubleshooting keyed to each workaround layer + sanity-check script - scripts/ccc/setup_podman.sh — one-shot podman/dbus/socket setup, builds a patched ubuntu:24.04 with ownership-wrapper shell scripts - scripts/ccc/run_ccc_smoke.sh — single-task LSF-ready smoke - scripts/ccc/run_ccc_experiment.sh — full cap-evolve run wrapper, supports --resume for walltime recovery - scripts/ccc/submit_ccc_experiment.sh — LSF bsub wrapper with sensible defaults for baseline vs full-iter jobs --- docs/RUN_ON_CCC.md | 604 +++++++++++++++++++++++++++ scripts/ccc/run_ccc_experiment.sh | 314 ++++++++++++++ scripts/ccc/run_ccc_smoke.sh | 212 ++++++++++ scripts/ccc/setup_podman.sh | 238 +++++++++++ scripts/ccc/submit_ccc_experiment.sh | 147 +++++++ 5 files changed, 1515 insertions(+) create mode 100644 docs/RUN_ON_CCC.md create mode 100755 scripts/ccc/run_ccc_experiment.sh create mode 100755 scripts/ccc/run_ccc_smoke.sh create mode 100755 scripts/ccc/setup_podman.sh create mode 100755 scripts/ccc/submit_ccc_experiment.sh diff --git a/docs/RUN_ON_CCC.md b/docs/RUN_ON_CCC.md new file mode 100644 index 00000000..7b6d8236 --- /dev/null +++ b/docs/RUN_ON_CCC.md @@ -0,0 +1,604 @@ +# Running Docker/Podman-based benchmarks on CCC (no admin) + +**Audience:** IBM CCC users who want to run tools that assume `docker` + +`docker compose` v2, on a locked-down cluster where you have no root, no +`sudo`, no `/etc/subuid` entry, and (on compute nodes) no systemd user +session. This document is a walkthrough of the exact set of userspace +workarounds we found through trial and error while getting BenchFlow + +SkillsBench + Claude Code running for the `cap-evolve` project. Every step +is reproducible without admin help. + +**Written:** 2026-07-29. **Updated:** 2026-07-30 with these iterative +fixes (each caught a specific class of failing task): +- v2 → v3: `chown`/`chgrp` wrappers (postinst chown failures) +- v3 → v4: `useradd`/`groupadd`/`usermod`/`groupmod`/`adduser`/`addgroup` + wrappers (packages that create system users) +- v4 → v5: `dpkg-statoverride` wrapper (dbus's setuid-helper ownership + fix, and everything downstream — libpam-systemd, gnumeric, libgtk, + libgoffice, libreoffice) +- PATH-ordering fix so our `docker` shim isn't shadowed by + `/usr/bin/docker` +- `poppler-utils` + `build-essential` preinstalled in the base image + +Environment: CCC RHEL 9.6, Podman 5.2.2 with podman-docker shim at +`/usr/bin/docker`, benchflow 0.6.5 installed via `uv tool`. + +--- + +## Why this is hard + +CCC's rootless podman is missing three things a typical Docker/podman +setup gives you for free: + +1. **A subuid range in `/etc/subuid`/`/etc/subgid`.** Standard rootless + setups have `yourname:100000:65536` — 65 536 UIDs to map into + containers. On CCC your user isn't listed, so the rootless namespace + only has UID 0 → your host UID. Every `useradd`/`chown`/setuid inside + a container to a different UID fails with `EINVAL`. + +2. **A systemd user session on compute nodes.** Login nodes have one, + compute nodes don't. Aardvark-dns (podman's DNS resolver for bridge + networks) needs systemd to spin up a transient scope; without it, + `podman-compose up` fails at container start. + +3. **A quiet `docker` command.** `/usr/bin/docker` on CCC is + `podman-docker`, which prints `Emulate Docker CLI using podman. + Create /etc/containers/nodocker to quiet msg.` to **stdout** on every + invocation. Tools that capture the stdout of `docker` commands end + up embedding that string in their results and downstream commands. + +All three have known workarounds. All three are userspace-only. + +--- + +## What you get + +After running the setup below, on any CCC node (login or compute) you can: + +```bash +source /scripts/ccc/setup_podman.sh # one line +docker run --rm ubuntu:24.04 uname -a # works +docker compose -f your.yaml up -d # works +bench eval run --sandbox docker ... # works +``` + +with no admin help. The setup script is idempotent — sourcing it again in +another shell is a no-op. + +--- + +## Prerequisites + +- CCC account with home in `/u/` and access to `/dccstor/...` for + shared data. Read/execute permission on + `/scripts/ccc/setup_podman.sh` (or copy it + to your own path). +- Podman 5.2+ and podman-docker on the host (default on current CCC). +- `dbus-daemon` in your `$PATH`. Anaconda's works + (`~/anaconda3/bin/dbus-daemon`) if you don't have it elsewhere. +- Python 3.10+ for anything you run on top. + +--- + +## One-time setup (do this once, persists in `$HOME`) + +### 1) Install Docker Compose v2 as a user CLI plugin + +Podman's default compose provider is `/usr/bin/podman-compose` (Python +1.5.0), which has a **different CLI** from Docker Compose v2 — it +doesn't accept `--project-directory` and other v2 flags that BenchFlow +(and most modern tooling) use. Install the real Compose v2 binary: + +```bash +mkdir -p ~/.docker/cli-plugins +curl -fsSL -o ~/.docker/cli-plugins/docker-compose \ + "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" +chmod +x ~/.docker/cli-plugins/docker-compose +~/.docker/cli-plugins/docker-compose version # should print "Docker Compose version vX.Y.Z" +``` + +### 2) Point podman at Compose v2 (and disable systemd/dbus paths) + +Write `~/.config/containers/containers.conf`: + +```bash +mkdir -p ~/.config/containers +cat > ~/.config/containers/containers.conf <<'EOF' +[containers] +# Compute nodes have no systemd user session, so avoid systemd/dbus paths. +cgroups = "disabled" + +[engine] +compose_providers = ["/u//.docker/cli-plugins/docker-compose"] +compose_warning_logs = false +# Same reason: don't call dbus, don't use systemd cgroup manager. +cgroup_manager = "cgroupfs" +events_logger = "file" +EOF +``` + +Replace `` with your username (the absolute path must resolve +on both login and compute nodes). + +### 3) Nothing else is one-time — the rest is done by `setup_podman.sh` per session. + +--- + +## Per-session setup (per compute node) + +Source `setup_podman.sh` at the start of every session on a fresh node. +The script is idempotent, so sourcing it in an existing session is fine +too. + +```bash +source /scripts/ccc/setup_podman.sh +``` + +You'll see: + +``` +[setup_podman] patched ubuntu:24.04 for apt-in-rootless (log: /tmp/podman-run-/patched-ubuntu-build.log) +[setup_podman] XDG_RUNTIME_DIR=/tmp/podman-run- +[setup_podman] graphroot=/tmp/podman- +[setup_podman] DOCKER_HOST=unix:///tmp/podman-run-/podman.sock +``` + +On the very first source on a fresh node, the "patched ubuntu:24.04" +step takes 2-3 minutes (see "What the script does" below). On subsequent +sources on the same node it's instant. + +### What the script does + +Read the file for the full detail — every action has a why. In brief: + +1. **Writes `~/.config/containers/storage.conf`** with `graphroot` and + `runroot` under `/tmp/podman-` and `/tmp/podman-run-` (host- + local, since GPFS can't hold overlay). Sets + `[storage.options.overlay] ignore_chown_errors = "true"` — this lets + the image unpack skip lchown calls that would need UID 42 (which we + don't have in our namespace). Without this, + `podman pull ubuntu:24.04` fails at `/etc/gshadow`. + +2. **Exports `XDG_RUNTIME_DIR=/tmp/podman-run-`.** On compute nodes + `/run/user/` doesn't exist (no systemd), and podman refuses to + start without a runtime dir. + +3. **Installs a userspace `docker` shim at `~/.local/bin/docker`** that + just execs `podman "$@"` with no chatty prefix. See "Why this is + hard" §3 above. Prepends `~/.local/bin` to `PATH` if missing. + +4. **Starts a private `dbus-daemon`** at + `$XDG_RUNTIME_DIR/dbus.sock`, exports + `DBUS_SESSION_BUS_ADDRESS=unix:path=...`. Some podman/netavark paths + assume a session bus is present even when we've configured them not + to. + +5. **Starts `podman system service`** at + `$XDG_RUNTIME_DIR/podman.sock`, exports + `DOCKER_HOST=unix://...`. This is the API socket Compose v2 talks to. + The script is idempotent — before starting, it kills any leftover + `podman system service` processes and their sockets. Without this + check, multiple sourcings ended up with 5 zombie services fighting + for the SQLite DB and producing "attempt to write a readonly + database" errors. + +6. **Builds a patched local `docker.io/library/ubuntu:24.04`** with + four groups of fixes baked in: + - `/etc/apt/apt.conf.d/00-rootless` sets + `APT::Sandbox::User "root";` (so apt doesn't try to setuid to + `_apt` = UID 42, which we can't map) and + `APT::Install-Recommends "false";` (so we don't pull in + `libc-devtools`/`libgd3`/`fontconfig-config` whose postinst scripts + do their own chown to non-root UIDs and fail). + - **Wraps a set of ownership/user-management binaries** to swallow + "Invalid argument" failures. Current list: + - `chown`, `chgrp` — postinst scripts do `chown fontconfig:root /...` + - `useradd`, `groupadd`, `usermod`, `groupmod`, `adduser`, + `addgroup` — packages that create system users (`_dbus`, + `messagebus`, `systemd-network`, `fontconfig`, etc.) + - `dpkg-statoverride` — dbus (and other packages) use it to set + the setuid bit on their launch helpers. It calls `fchown()` via + libc, so wrapping shell-level `chown` alone doesn't cover it. + + Each wrapper is a 2-line `sh` script that calls the real binary + (preserved at `.real`) and returns 0 regardless of its + exit code. Files still get created, just without correct + ownership — usually fine, since the container's rootless namespace + doesn't enforce those UIDs anyway. + - Pre-installs `python3 python3-pip curl poppler-utils build-essential` + so downstream Dockerfiles that install these packages find them + already present and get a fast no-op. `build-essential` is + ~500 MB but pays for itself across tasks: multiple SkillsBench + Dockerfiles pull it in transitively, and the tail of failing + postinst scripts it drags in is the biggest source of image-build + failures on CCC. + + Any downstream `FROM ubuntu:24.04` picks up our patched local image + (podman uses local before remote when tags match). + +7. **Prepends `~/.local/bin` to `$PATH`** — not just "adds if missing" + but explicitly puts it FIRST. If a login script has already added + `~/.local/bin` to a later position, our `docker` shim (§3) would be + shadowed by `/usr/bin/docker`, and every rollout would inherit the + "Emulate Docker CLI..." message → garbled `-w` arg → `rc=127` at + agent exec. This bug ate a whole 30-minute baseline before we + spotted the PATH ordering issue. + +--- + +## For BenchFlow-based tooling (SkillsBench, etc.) specifically + +Two more patches are needed to bench itself. These affect `bench`'s +site-packages (`~/.local/share/uv/tools/benchflow/lib/.../benchflow/`); +they survive across sessions but are lost on `uv tool install --force`. + +### A) Force host networking in every task container + +BenchFlow's compose base yaml sets up a bridge network by default. +Bridge → netavark → aardvark-dns → systemd → fail on compute nodes. +Force host networking so aardvark isn't invoked at all: + +```bash +python3 - <<'PY' +import shutil +p = "/u//.local/share/uv/tools/benchflow/lib/python3.12/site-packages/benchflow/sandbox/_compose_files/docker-compose-base.yaml" +if not __import__("os").path.exists(p + ".orig"): + shutil.copy(p, p + ".orig") +s = open(p).read() +old = 'services:\n main:\n labels:' +new = 'services:\n main:\n # CCC workaround: skip bridge network; aardvark-dns needs systemd\n network_mode: host\n labels:' +if old in s: + open(p, "w").write(s.replace(old, new)) + print("patched: network_mode: host") +else: + print("already patched or file changed") +PY +``` + +**Trade-off:** the container shares the host's network stack. That means +no port isolation and no per-container hostname. Fine for tasks that +don't listen on ports (which is 99% of SkillsBench). + +### B) Bypass `docker compose cp` (which fails on our UID namespace) + +`docker compose cp` preserves the source file's ownership when copying +between host and container. Since our host UIDs (561567:608693 in my +case) don't exist inside the container's rootless namespace, every +`cp` blows up on `lchown`. The fix: replace bench's cp-based upload and +download with `docker compose exec -T` piped through `tar`, so the +container-side tar (running as root) owns everything. + +Apply the upload patch: + +```bash +python3 - <<'PY' +import re +p = "/u//.local/share/uv/tools/benchflow/lib/python3.12/site-packages/benchflow/sandbox/docker.py" +s = open(p).read() + +helper = ''' async def _upload_via_exec(self, source_path, target_path: str, is_dir: bool) -> None: + """CCC workaround: docker compose cp preserves source UIDs which fail + to lchown inside rootless-podman (no /etc/subuid entry). We bypass cp + by streaming the file/dir into the container via `exec -T` + stdin — + the exec process runs as root, so files land as root:root and no + cross-namespace chown is attempted. + """ + import io as _io + import tarfile as _tar + buf = _io.BytesIO() + def _root_owned(ti): + ti.uid = 0; ti.gid = 0 + ti.uname = "root"; ti.gname = "root" + return ti + if is_dir: + with _tar.open(fileobj=buf, mode="w") as tf: + tf.add(str(source_path), arcname=".", filter=_root_owned) + remote_cmd = f"tar xf - --no-same-owner -C {shlex.quote(target_path)}" + else: + with open(source_path, "rb") as f: + data = f.read() + with _tar.open(fileobj=buf, mode="w") as tf: + info = _tar.TarInfo(name="_upload") + info.size = len(data) + info.mode = 0o644 + info.uid = 0; info.gid = 0 + info.uname = "root"; info.gname = "root" + tf.addfile(info, _io.BytesIO(data)) + remote_cmd = ( + f"tar xf - --no-same-owner -C /tmp && " + f"mv /tmp/_upload {shlex.quote(target_path)}" + ) + full_command = [ + "docker", "compose", "--project-name", + _sanitize_docker_compose_project_name(self.session_id), + "--project-directory", + str(self.environment_dir.resolve().absolute()), + ] + for path in self._docker_compose_paths: + full_command.extend(["-f", str(path.resolve().absolute())]) + full_command.extend(["exec", "-T", "main", "sh", "-c", remote_cmd]) + env = self._docker_compose_env() + process = await asyncio.create_subprocess_exec( + *full_command, + env=env, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + ) + stdout_bytes, _ = await process.communicate(input=buf.getvalue()) + if process.returncode != 0: + raise RuntimeError( + f"upload_via_exec failed (rc={process.returncode}): " + f"{stdout_bytes.decode(errors='replace')}" + ) + + async def upload_file''' + +old_upload_file = ''' async def upload_file(self, source_path: Path | str, target_path: str) -> None: + target_parent = str(Path(target_path).parent) + if target_parent not in {"", "."}: + await self.exec(f"mkdir -p {shlex.quote(target_parent)}", user="root") + await self._run_docker_compose_command( + ["cp", str(source_path), f"main:{target_path}"], + check=True, + )''' +new_upload_file = ''' async def upload_file(self, source_path: Path | str, target_path: str) -> None: + target_parent = str(Path(target_path).parent) + if target_parent not in {"", "."}: + await self.exec(f"mkdir -p {shlex.quote(target_parent)}", user="root") + await self._upload_via_exec(source_path, target_path, is_dir=False)''' + +if "_upload_via_exec" not in s: + assert old_upload_file in s + s = s.replace(old_upload_file, helper.replace(" async def upload_file", "").rstrip() + "\n\n" + new_upload_file, 1) + old_dir_call = ''' await self._run_docker_compose_command( + ["cp", f"{source_dir}/.", f"{service}:{target_dir}"], + check=True, + )''' + new_dir_call = ''' await self._upload_via_exec(source_dir, target_dir, is_dir=True)''' + s = s.replace(old_dir_call, new_dir_call, 1) + open(p, "w").write(s) + print("upload patch applied") +else: + print("upload patch already applied") +PY +``` + +Apply the download patch (mirrors the upload — pipe tar from container +stdout to host): + +```bash +python3 - <<'PY' +p = "/u//.local/share/uv/tools/benchflow/lib/python3.12/site-packages/benchflow/sandbox/docker.py" +s = open(p).read() + +helper = ''' async def _download_via_exec(self, source_path: str, target_path, is_dir: bool, service: str = "main") -> None: + """CCC workaround: reverse of _upload_via_exec. Stream a tar of the + container-side path to our stdout, unpack into the host target dir. + """ + import io as _io + import os as _os + import tarfile as _tar + if is_dir: + remote_cmd = f"tar cf - -C {shlex.quote(source_path)} ." + _os.makedirs(str(target_path), exist_ok=True) + else: + parent = str(Path(source_path).parent) or "/" + name = Path(source_path).name + remote_cmd = f"tar cf - -C {shlex.quote(parent)} {shlex.quote(name)}" + full_command = [ + "docker", "compose", "--project-name", + _sanitize_docker_compose_project_name(self.session_id), + "--project-directory", + str(self.environment_dir.resolve().absolute()), + ] + for path in self._docker_compose_paths: + full_command.extend(["-f", str(path.resolve().absolute())]) + full_command.extend(["exec", "-T", service, "sh", "-c", remote_cmd]) + env = self._docker_compose_env() + process = await asyncio.create_subprocess_exec( + *full_command, + env=env, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout_bytes, stderr_bytes = await process.communicate() + if process.returncode != 0: + raise RuntimeError( + f"download_via_exec failed (rc={process.returncode}): " + f"{stderr_bytes.decode(errors='replace')}" + ) + buf = _io.BytesIO(stdout_bytes) + with _tar.open(fileobj=buf, mode="r|") as tf: + if is_dir: + tf.extractall(str(target_path), filter="data") + else: + import tempfile as _tempfile, shutil as _shutil + with _tempfile.TemporaryDirectory() as td: + tf.extractall(td, filter="data") + src_extracted = _os.path.join(td, Path(source_path).name) + _shutil.move(src_extracted, str(target_path)) + + async def download_file''' + +old_download_file = ''' async def download_file(self, source_path: str, target_path: Path | str) -> None: + await self._chown_to_host_user(source_path) + await self._run_docker_compose_command( + ["cp", f"main:{source_path}", str(target_path)], + check=True, + )''' +new_download_file = ''' async def download_file(self, source_path: str, target_path: Path | str) -> None: + await self._download_via_exec(source_path, target_path, is_dir=False)''' + +old_download_dir = ''' await self._chown_to_host_user(source_dir, recursive=True, service=service) + await self._run_docker_compose_command( + ["cp", f"{service}:{source_dir}/.", str(target_dir)], + check=True, + )''' +new_download_dir = ''' await self._download_via_exec(source_dir, target_dir, is_dir=True, service=service)''' + +if "_download_via_exec" not in s: + assert old_download_file in s + assert old_download_dir in s + s = s.replace(old_download_file, helper.replace(" async def download_file", "").rstrip() + "\n\n" + new_download_file, 1) + s = s.replace(old_download_dir, new_download_dir, 1) + open(p, "w").write(s) + print("download patch applied") +else: + print("download patch already applied") +PY +``` + +### C) Pass `--sandbox-user ''` to `bench eval run` + +BenchFlow defaults to running the agent inside the container as user +`agent`. Creating that user requires `useradd -m` which needs UIDs we +don't have. Run as root instead: + +```bash +bench eval run --sandbox-user '' ... # NOTE the empty string +``` + +If you're driving bench through cap-evolve, the adapter should read +`SKILLSBENCH_SANDBOX_USER` from `.env` and pass it through: + +```bash +# .env +SKILLSBENCH_SANDBOX_USER= +``` + +--- + +## Verification: run the smoke + +```bash +# Set up +source /scripts/ccc/setup_podman.sh + +# From your intake worktree (or any dir with a valid .env) +cd .../intake_skillbench_c1 +set -a; source ./.env; set +a + +# One task, one rollout — takes ~4 min +rm -rf /tmp/skillsbench-smoke-claude +bench eval run \ + --tasks-dir "$SKILLSBENCH_TASKS_DIR" \ + --include offer-letter-generator \ + --agent claude-agent-acp --model claude-opus-4-6 \ + --sandbox docker \ + --sandbox-user '' \ + --skill-mode with-skill \ + --skills-dir "$PWD/.capevolve/project/seed_capability" \ + --jobs-dir /tmp/skillsbench-smoke-claude \ + --agent-env "ANTHROPIC_BASE_URL=${ANTHROPIC_BASE_URL:?}" \ + --agent-env "ANTHROPIC_AUTH_TOKEN=${ANTHROPIC_AUTH_TOKEN:?}" +``` + +Success looks like: + +``` +✓ 1 passed ✗ 0 failed ⚠ 0 errored +``` + +or + +``` +✓ 0 passed ✗ 1 failed ⚠ 0 errored +``` + +Both mean the plumbing works. `failed` just means the agent's output +didn't match the verifier's expectation on this particular task — the +whole stack (container, agent install, agent execution, verifier) +worked end-to-end. + +`errored: 1` means one of the workarounds above is missing. See the +troubleshooting table. + +--- + +## Troubleshooting: which layer failed? + +Read the failing rollout's `result.json` under `/tmp/skillsbench-smoke-*/ +/__/result.json`. The `error` field is verbose; +match against these patterns: + +| Error contains | Which workaround failed | +|---|---| +| `insufficient UIDs or GIDs available in user namespace` at image unpack | Storage's `ignore_chown_errors` — check `~/.config/containers/storage.conf`. | +| `setuid 42 failed` / `Method http has died` during `apt-get install` | Patched ubuntu:24.04 not present — check `podman images` for it, or re-source `setup_podman.sh`. | +| `libc-devtools ... dependency problems` | `--no-install-recommends` in the patched image; re-check the image's `/etc/apt/apt.conf.d/00-rootless`. | +| `Errors were encountered while processing: fontconfig-config / libcairo2 / poppler-utils / ...` | chown/chgrp wrapper not present in the patched base — rebuild with `podman rmi -f docker.io/library/ubuntu:24.04 && rm /tmp/podman-/.patched-ubuntu24-* && source setup_podman.sh`. Verify with `podman run --rm docker.io/library/ubuntu:24.04 chown nobody:nobody /tmp && echo ok`. | +| `dpkg-statoverride: error: error setting ownership of ... : Invalid argument` (dbus/libpam-systemd/gnumeric/libgtk/libgoffice/libreoffice cascade) | `dpkg-statoverride` wrapper not present in patched base — this is the v5 fix. Rebuild the patched base (same recipe as above; setup_podman.sh v4+ includes the wrapper). Verify with `podman run --rm docker.io/library/ubuntu:24.04 head -2 /usr/bin/dpkg-statoverride` — should print `#!/bin/sh` + the swallow-error line. | +| `aardvark-dns failed to start ... systemd1` | `network_mode: host` patch (§A above) not applied to base compose yaml. | +| `Failed to connect to bus` | dbus-daemon not running — check `pgrep dbus-daemon`. | +| `readonly database` | Zombie podman services — `pkill -f 'podman system service'` and re-source. | +| `copier: put: error setting ownership of ... to 561567:...` | Upload patch (§B) not applied. | +| `chown: changing ownership of '/home/agent/.claude': Invalid argument` | `--sandbox-user ''` (§C) not passed. | +| `Agent claude-agent-acp install failed (rc=127)` with `Could not resolve host` | `network_mode: host` patch not effective — container has no network. | +| `crun: /root...` in exec | Docker-shim not on PATH — bench's stdout probe captured the "Emulate Docker CLI..." message. Check `which docker` (must be `~/.local/bin/docker`, NOT `/usr/bin/docker`). Most common cause: `~/.local/bin` was already in `$PATH` but AFTER `/usr/bin`. Fix by re-sourcing `setup_podman.sh` (v3+ prepends explicitly). | + +--- + +## What's NOT solved + +- **Task images that need to `useradd` or `chown` to arbitrary UIDs at + runtime** will still fail. You'd hit this on any task whose Dockerfile + installs software that assumes non-root operation (nginx, postgres, …). + For SkillsBench (office-doc tasks), this hasn't come up. + +- **Running `bench --sandbox modal` from ACP-agent tasks.** Benchflow + 0.6.5 has a bug where it dispatches Modal sandboxes through the Daytona + process class (`sandbox.process.exec()` on Modal's Sandbox — no such + attribute). This is a separate benchflow issue, unrelated to the CCC + workarounds above. If you want to use Modal, you'd need to author a + `ModalProcess` class (~200 LOC). + +- **`uv tool install --force benchflow`** will wipe out patches (§A, §B). + If you upgrade, re-apply them from this document. + +--- + +## Sanity check: everything installed? + +```bash +[ -x ~/.docker/cli-plugins/docker-compose ] && echo "compose v2: OK" || echo "compose v2: MISSING" +[ -f ~/.config/containers/containers.conf ] && echo "containers.conf: OK" || echo "containers.conf: MISSING" +[ -x ~/.local/bin/docker ] && echo "docker shim: OK" || echo "docker shim: MISSING (setup_podman.sh writes this)" +[ "$(readlink -f "$(which docker)")" = "$HOME/.local/bin/docker" ] && echo "docker shim WINS PATH: OK" || echo "docker shim SHADOWED: FAIL (/usr/bin/docker still first)" +podman images docker.io/library/ubuntu:24.04 | grep -q ubuntu && echo "patched ubuntu: OK" || echo "patched ubuntu: MISSING (setup_podman.sh builds this)" +podman run --rm docker.io/library/ubuntu:24.04 chown nobody:nobody /tmp 2>/dev/null && echo "chown wrapper: OK" || echo "chown wrapper: MISSING (rebuild patched base)" +podman run --rm docker.io/library/ubuntu:24.04 useradd _dbus 2>/dev/null && echo "useradd wrapper: OK" || echo "useradd wrapper: MISSING (rebuild patched base with v4+)" +podman run --rm docker.io/library/ubuntu:24.04 sh -c '[ -x /usr/bin/dpkg-statoverride.real ]' && echo "dpkg-statoverride wrapper: OK" || echo "dpkg-statoverride wrapper: MISSING (rebuild patched base with v5+)" +podman run --rm docker.io/library/ubuntu:24.04 sh -c 'command -v pdfinfo gcc >/dev/null' && echo "heavy preinstalls: OK" || echo "heavy preinstalls: MISSING (rebuild patched base)" +grep -q "network_mode: host" /u/$USER/.local/share/uv/tools/benchflow/lib/python*/site-packages/benchflow/sandbox/_compose_files/docker-compose-base.yaml 2>/dev/null && echo "bench yaml patch (§A): OK" || echo "bench yaml patch (§A): MISSING" +grep -q "_upload_via_exec" /u/$USER/.local/share/uv/tools/benchflow/lib/python*/site-packages/benchflow/sandbox/docker.py 2>/dev/null && echo "bench upload patch (§B): OK" || echo "bench upload patch (§B): MISSING" +grep -q "_download_via_exec" /u/$USER/.local/share/uv/tools/benchflow/lib/python*/site-packages/benchflow/sandbox/docker.py 2>/dev/null && echo "bench download patch (§B): OK" || echo "bench download patch (§B): MISSING" +``` + +If any prints MISSING or FAIL, re-run the corresponding step above. The +`docker shim WINS PATH` line is a common gotcha: the file can exist but +be shadowed by an earlier PATH entry — that's a rebuild-the-baseline +disaster. + +--- + +## Batch (LSF) mode — TODO + +Running the cap-evolve baseline through `bsub` on CCC needs one more +consideration: the interactive-shell setup we just did (source +`setup_podman.sh` and start dbus/podman services) needs to be part of +the batch job's environment. Draft plan (untested): + +1. Wrap the workload in a shell script that sources `setup_podman.sh` + first, then runs `cap-evolve run ...`. +2. `bsub` with enough disk on `/tmp` for the image cache (~200 MB for + ubuntu:24.04 + task images), enough memory for concurrent rollouts + (~4 GB × concurrency), and enough wall time (7-iter run is 4-6h). +3. LSF may kill the podman service on job teardown; if that leaves + half-written state in `/tmp`, `setup_podman.sh` should handle the + next run's cleanup, but verify. +4. `DOCKER_HOST` and `DBUS_SESSION_BUS_ADDRESS` are per-user paths in + `$XDG_RUNTIME_DIR` (host-local `/tmp`) — safe. + +Detailed instructions will follow after we've done a batch dry-run. diff --git a/scripts/ccc/run_ccc_experiment.sh b/scripts/ccc/run_ccc_experiment.sh new file mode 100755 index 00000000..1010130a --- /dev/null +++ b/scripts/ccc/run_ccc_experiment.sh @@ -0,0 +1,314 @@ +#!/bin/bash +# +# Run one cap-evolve experiment on a CCC compute node. +# +# Designed to be called via `bsub` (see submit_ccc_experiment.sh) or run +# directly on a compute node for testing. Handles all the CCC-specific +# podman setup and saves results under the project tree. +# +# Usage: +# # Direct execution (on a compute node with an interactive shell) +# bash scripts/ccc/run_ccc_experiment.sh \ +# --suite-id baseline_v1 \ +# --max-iterations 0 +# +# # Baseline (iter=0) at a specific run-id +# bash scripts/ccc/run_ccc_experiment.sh \ +# --suite-id baseline_v1 \ +# --run-id my_test \ +# --max-iterations 0 +# +# # Full 7-iter run +# bash scripts/ccc/run_ccc_experiment.sh \ +# --suite-id iter7_opus_v1 \ +# --max-iterations 7 +# +# Output layout under the project (all under $PROJECT_ROOT/results/): +# +# results/ +# / # e.g. "baseline_v1" +# / # e.g. LSF job id, or timestamp when local +# setup.log # setup_podman.sh output +# cap-evolve.log # cap-evolve stdout+stderr +# env_snapshot.txt # .env + capevolve.yaml + git commit + hostname +# run/ # cap-evolve's run dir (symlinked from .capevolve/run_) +# +# Suite-id / run-id conventions: +# - suite-id: your logical grouping ("baseline_ccc_v1", "iter7_opus_20260730"). +# Not required to be unique across sessions; multiple runs can share. +# - run-id: uniquely identifies THIS run. Defaults to $LSB_JOBID (when +# submitted via LSF) or local_ otherwise. +# +# The script exits non-zero if setup fails or cap-evolve returns an error. + +set -eo pipefail + +# -------------------------------------------------------------------- +# Argument parsing +# -------------------------------------------------------------------- +SUITE_ID="" +RUN_ID="" +RUN_TS="" # stable name for cap-evolve's run dir; defaults to RUN_ID +RESUME=false # pass --resume to cap-evolve (continue an interrupted run) +MAX_ITERATIONS="0" +SPEC=".capevolve/project/capevolve.yaml" +PROJECT_DIR=".capevolve/project" +INCLUDE="" # optional; forwarded as --include to bench if bench-mode is added later +EXTRA_ARGS="" # any extra flags to append verbatim to cap-evolve +DRY_RUN=false + +usage() { + sed -n '2,45p' "$0" + exit 2 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --suite-id) SUITE_ID="$2"; shift 2 ;; + --run-id) RUN_ID="$2"; shift 2 ;; + --run-ts) RUN_TS="$2"; shift 2 ;; + --resume) RESUME=true; shift ;; + --max-iterations) MAX_ITERATIONS="$2"; shift 2 ;; + --spec) SPEC="$2"; shift 2 ;; + --project) PROJECT_DIR="$2"; shift 2 ;; + --extra-args) EXTRA_ARGS="$2"; shift 2 ;; + --dry-run) DRY_RUN=true; shift ;; + -h|--help) usage ;; + *) echo "Unknown option: $1" >&2; usage ;; + esac +done + +# --run-ts defaults to RUN_ID when not explicitly given. Using the same +# --run-ts across LSF jobs is how you resume: cap-evolve looks up its state +# under .capevolve/run_/, so a stable name lets the second submission +# find the first's checkpoint. +if [[ -z "$RUN_TS" ]]; then + RUN_TS="$RUN_ID" +fi + +if [[ -z "$SUITE_ID" ]]; then + echo "ERROR: --suite-id is required (e.g. --suite-id baseline_v1)" >&2 + usage +fi + +# Default RUN_ID from LSF job id, else a timestamp. +if [[ -z "$RUN_ID" ]]; then + if [[ -n "${LSB_JOBID:-}" ]]; then + RUN_ID="$LSB_JOBID" + else + RUN_ID="local_$(date +%Y%m%d_%H%M%S)" + fi +fi + +# RUN_TS still empty means the user didn't pass --run-ts and RUN_ID was +# resolved above. Re-default RUN_TS to RUN_ID now that RUN_ID is known. +if [[ -z "$RUN_TS" ]]; then + RUN_TS="$RUN_ID" +fi + +# -------------------------------------------------------------------- +# Locate the project root +# -------------------------------------------------------------------- +# The intake worktree the script lives in. Resolve by climbing up from +# this file's dir. Users can override with $PROJECT_ROOT. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [[ -z "${PROJECT_ROOT:-}" ]]; then + PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +fi + +if [[ ! -f "$PROJECT_ROOT/$SPEC" ]]; then + echo "ERROR: --spec not found: $PROJECT_ROOT/$SPEC" >&2 + echo " (try running from the intake worktree, or set PROJECT_ROOT)" >&2 + exit 2 +fi + +# -------------------------------------------------------------------- +# Output directory +# -------------------------------------------------------------------- +OUT_DIR="$PROJECT_ROOT/results/$SUITE_ID/$RUN_ID" +mkdir -p "$OUT_DIR" + +SETUP_LOG="$OUT_DIR/setup.log" +RUN_LOG="$OUT_DIR/cap-evolve.log" +ENV_SNAP="$OUT_DIR/env_snapshot.txt" + +# -------------------------------------------------------------------- +# Print banner +# -------------------------------------------------------------------- +banner() { + printf '\n============================================================\n' + printf '%s\n' "$1" + printf '============================================================\n' +} + +banner "cap-evolve on CCC: $SUITE_ID / $RUN_ID" +{ + echo "Host: $(hostname)" + echo "Start: $(date -Iseconds)" + echo "PROJECT_ROOT: $PROJECT_ROOT" + echo "OUT_DIR: $OUT_DIR" + echo "SPEC: $SPEC" + echo "PROJECT_DIR: $PROJECT_DIR" + echo "RUN_TS: $RUN_TS (cap-evolve run dir → .capevolve/run_$RUN_TS/)" + echo "RESUME: $RESUME" + echo "MAX_ITER: $MAX_ITERATIONS" + echo "EXTRA_ARGS: $EXTRA_ARGS" + echo "DRY_RUN: $DRY_RUN" + if [[ -n "${LSB_JOBID:-}" ]]; then + echo "LSF job: $LSB_JOBID" + echo "LSF queue: ${LSB_QUEUE:-}" + echo "LSF host: ${LSB_HOSTS:-}" + fi +} | tee "$ENV_SNAP" + +if [[ "$DRY_RUN" == true ]]; then + echo + echo "(dry-run: exiting before setup)" + exit 0 +fi + +# -------------------------------------------------------------------- +# Phase 1: CCC podman setup (rootless podman + docker shim + patched +# base image + bench patches, all done idempotently) +# -------------------------------------------------------------------- +banner "Phase 1: setup_podman.sh" +# CRITICAL: `source X | tee Y` puts X in a subshell — its `export`s +# would be discarded. Process substitution `> >(tee ...)` keeps `source` +# in the current shell so PATH/DOCKER_HOST/... persist. +# shellcheck disable=SC1091 +source "$SCRIPT_DIR/setup_podman.sh" \ + > >(tee "$SETUP_LOG") 2>&1 +# Wait a tick so tee finishes flushing to disk before we proceed. +wait 2>/dev/null || true + +# Belt-and-suspenders: prepend the docker shim explicitly, in case +# setup_podman.sh's PATH prepend logic regresses. +if [[ -x "$HOME/.local/bin/docker" ]]; then + export PATH="$HOME/.local/bin:${PATH/#$HOME\/.local\/bin:/}" +fi + +# Sanity: docker shim must win the PATH lookup, otherwise we'll get the +# "Emulate Docker CLI using podman" stdout pollution in every bench probe. +if [[ "$(readlink -f "$(which docker)")" != "$HOME/.local/bin/docker" ]]; then + echo "FATAL: 'docker' resolves to $(which docker), not $HOME/.local/bin/docker." >&2 + echo " PATH ordering is wrong; setup_podman.sh should have prepended it." >&2 + echo " Aborting to avoid a corrupted run." >&2 + exit 3 +fi + +# -------------------------------------------------------------------- +# Phase 2: load credentials from .env +# -------------------------------------------------------------------- +banner "Phase 2: loading .env" +if [[ ! -f "$PROJECT_ROOT/.env" ]]; then + echo "FATAL: $PROJECT_ROOT/.env not found." >&2 + exit 2 +fi +set -a +# shellcheck disable=SC1090,SC1091 +source "$PROJECT_ROOT/.env" +set +a +: "${ANTHROPIC_BASE_URL:?ANTHROPIC_BASE_URL missing from .env}" +: "${ANTHROPIC_AUTH_TOKEN:?ANTHROPIC_AUTH_TOKEN missing from .env}" +: "${SKILLSBENCH_TASKS_DIR:?SKILLSBENCH_TASKS_DIR missing from .env}" + +# Snapshot the exact resolved config alongside the results. +{ + echo + echo "===== .env (redacted) =====" + sed 's/\(TOKEN\|KEY\)=.*/\1=/' "$PROJECT_ROOT/.env" + echo + echo "===== $SPEC =====" + cat "$PROJECT_ROOT/$SPEC" + echo + echo "===== git status (worktree) =====" + ( cd "$PROJECT_ROOT" && git log -1 --format='commit %H%n%s%n%n%b' 2>&1 || echo "(not a git repo)" ) + echo + echo "===== podman info =====" + podman info 2>&1 | head -60 +} >> "$ENV_SNAP" + +# -------------------------------------------------------------------- +# Phase 3: cap-evolve run +# -------------------------------------------------------------------- +banner "Phase 3: cap-evolve run" +cd "$PROJECT_ROOT" + +# Cap-evolve writes to .capevolve/run_/ by default. Symlink that +# under results///run/ so the run's artifacts live under +# the project tree without cap-evolve needing to know about our layout. +CE_RUN_DIRNAME="run_${RUN_TS}" +CE_RUN_ABS="$PROJECT_ROOT/.capevolve/$CE_RUN_DIRNAME" +LINK_TARGET="$OUT_DIR/run" +ln -sfn "$CE_RUN_ABS" "$LINK_TARGET" + +export PYTHONPATH="$PROJECT_DIR/adapters" +export CAPEVOLVE_SKILLS_DIR="$PROJECT_ROOT/skills" + +# cap-evolve CLI. Override with $CAP_EVOLVE_BIN, else use whatever's in PATH. +# In a typical install `pip install -e ./core` puts a `cap-evolve` script in +# the active venv's bin/; make sure that venv is on PATH before invoking, or +# set CAP_EVOLVE_BIN to the absolute path. +CE_BIN="${CAP_EVOLVE_BIN:-$(command -v cap-evolve || true)}" +if [[ -z "$CE_BIN" || ! -x "$CE_BIN" ]]; then + echo "FATAL: cap-evolve CLI not found." >&2 + echo " Either activate the venv where you installed cap-evolve," >&2 + echo " or export CAP_EVOLVE_BIN=/absolute/path/to/cap-evolve" >&2 + exit 2 +fi + +# Compose the command line +CE_CMD=( + "$CE_BIN" run + --spec "$SPEC" + --project "$PROJECT_DIR" + --run-ts "$RUN_TS" + --max-iterations "$MAX_ITERATIONS" +) +if [[ "$RESUME" == true ]]; then + CE_CMD+=(--resume) +fi +if [[ -n "$EXTRA_ARGS" ]]; then + # shellcheck disable=SC2206 + CE_CMD+=($EXTRA_ARGS) +fi + +echo "Running: ${CE_CMD[*]}" +echo + +# Run under `stdbuf` so log lines flush live to the file — useful when +# tailing $RUN_LOG from another shell mid-experiment. +set +e +stdbuf -oL -eL "${CE_CMD[@]}" 2>&1 | tee "$RUN_LOG" +RC="${PIPESTATUS[0]}" +set -e + +# -------------------------------------------------------------------- +# Phase 4: summarize + exit +# -------------------------------------------------------------------- +banner "Phase 4: done" +echo "End: $(date -Iseconds)" +echo "Exit: $RC" +echo "Results: $OUT_DIR" +echo " setup: $SETUP_LOG" +echo " run: $RUN_LOG" +echo " env: $ENV_SNAP" +echo " cap-evolve run dir (symlink → $CE_RUN_ABS): $LINK_TARGET" + +# If we produced a baseline.json, show its headline +if [[ -f "$CE_RUN_ABS/baseline.json" ]]; then + echo + echo "===== baseline.json headline =====" + python3 -c " +import json +d = json.load(open('$CE_RUN_ABS/baseline.json')) +v = d.get('val', {}) +print(f\"val reward: {v.get('reward')} stderr: {v.get('stderr')}\") +print(f\"pass_at_k: {v.get('pass_at_k')}\") +print(f\"per_task:\") +for t in v.get('per_task', []): + print(f\" {t.get('task_id'):<28} reward={t.get('reward')} stderr={t.get('stderr')} trial_rewards={t.get('trial_rewards')}\") +" || echo "(couldn't parse baseline.json)" +fi + +exit "$RC" diff --git a/scripts/ccc/run_ccc_smoke.sh b/scripts/ccc/run_ccc_smoke.sh new file mode 100755 index 00000000..7f88b6ba --- /dev/null +++ b/scripts/ccc/run_ccc_smoke.sh @@ -0,0 +1,212 @@ +#!/bin/bash +# +# Smoke-test the CCC batch environment on a SINGLE task + SINGLE trial. +# +# Cheap (~10 min, few cents) way to prove the LSF path — same podman +# setup, same bench command shape, same result-tree layout — before +# committing to a full cap-evolve baseline. +# +# Usage (direct, on an interactive compute node): +# bash scripts/ccc/run_ccc_smoke.sh --suite-id smoke_batch_v1 +# +# Usage (batch, via LSF): +# bsub -q normal -M 100G -n 1 \ +# -oo $HOME/ccc_logs/%J.stdout \ +# -eo $HOME/ccc_logs/%J.stderr \ +# bash scripts/ccc/run_ccc_smoke.sh --suite-id smoke_batch_v1 +# +# Options: +# --suite-id ID [REQUIRED] result grouping (e.g. "smoke_batch_v1") +# --run-id ID unique-within-suite; defaults to $LSB_JOBID or local_ +# --task NAME SkillsBench task id (default: invoice-fraud-detection — +# the one we know passes with seed skills) +# +# Output layout: +# results/// +# setup.log setup_podman.sh output +# bench.log bench eval run stdout+stderr +# env_snapshot.txt .env(redacted) + git commit + podman info +# bench_jobs/ bench's --jobs-dir output +# PASS or FAIL or ERROR marker file with the outcome + +set -eo pipefail + +SUITE_ID="" +RUN_ID="" +TASK="invoice-fraud-detection" + +while [[ $# -gt 0 ]]; do + case "$1" in + --suite-id) SUITE_ID="$2"; shift 2 ;; + --run-id) RUN_ID="$2"; shift 2 ;; + --task) TASK="$2"; shift 2 ;; + -h|--help) sed -n '2,25p' "$0"; exit 2 ;; + *) echo "Unknown option: $1" >&2; exit 2 ;; + esac +done + +if [[ -z "$SUITE_ID" ]]; then + echo "ERROR: --suite-id is required" >&2 + exit 2 +fi + +if [[ -z "$RUN_ID" ]]; then + if [[ -n "${LSB_JOBID:-}" ]]; then + RUN_ID="$LSB_JOBID" + else + RUN_ID="local_$(date +%Y%m%d_%H%M%S)" + fi +fi + +# Locate project root +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="${PROJECT_ROOT:-$(cd "$SCRIPT_DIR/../.." && pwd)}" + +OUT_DIR="$PROJECT_ROOT/results/$SUITE_ID/$RUN_ID" +mkdir -p "$OUT_DIR" + +SETUP_LOG="$OUT_DIR/setup.log" +BENCH_LOG="$OUT_DIR/bench.log" +ENV_SNAP="$OUT_DIR/env_snapshot.txt" + +banner() { printf '\n============================================================\n%s\n============================================================\n' "$1"; } + +banner "cap-evolve CCC smoke: $SUITE_ID / $RUN_ID (task=$TASK)" +{ + echo "Host: $(hostname)" + echo "Start: $(date -Iseconds)" + echo "PROJECT_ROOT: $PROJECT_ROOT" + echo "OUT_DIR: $OUT_DIR" + echo "TASK: $TASK" + [[ -n "${LSB_JOBID:-}" ]] && echo "LSF job: $LSB_JOBID queue=${LSB_QUEUE:-} host=${LSB_HOSTS:-}" +} | tee "$ENV_SNAP" + +# --- Phase 1: podman setup --- +banner "Phase 1: setup_podman.sh" +# CRITICAL: `source X | tee Y` runs X in a subshell — its `export`s +# would be discarded. Process substitution `> >(tee ...)` keeps `source` +# in the current shell so PATH/DOCKER_HOST/... persist. +# shellcheck disable=SC1091 +source "$SCRIPT_DIR/setup_podman.sh" \ + > >(tee "$SETUP_LOG") 2>&1 +# Wait a tick so tee finishes flushing to disk before we proceed. +wait 2>/dev/null || true + +# Belt-and-suspenders: prepend the docker shim explicitly, in case +# setup_podman.sh's PATH prepend logic regresses. +if [[ -x "$HOME/.local/bin/docker" ]]; then + export PATH="$HOME/.local/bin:${PATH/#$HOME\/.local\/bin:/}" +fi + +# Sanity: docker shim wins PATH? +if [[ "$(readlink -f "$(which docker)")" != "$HOME/.local/bin/docker" ]]; then + echo "FATAL: 'docker' resolves to $(which docker), not $HOME/.local/bin/docker." + echo " PATH ordering is wrong; the run would be corrupted. Aborting." + echo "ERROR" > "$OUT_DIR/OUTCOME" + exit 3 +fi + +# --- Phase 2: load .env --- +banner "Phase 2: loading .env" +if [[ ! -f "$PROJECT_ROOT/.env" ]]; then + echo "FATAL: $PROJECT_ROOT/.env not found." + echo "ERROR" > "$OUT_DIR/OUTCOME" + exit 2 +fi +set -a +# shellcheck disable=SC1090 +source "$PROJECT_ROOT/.env" +set +a +: "${ANTHROPIC_BASE_URL:?}" +: "${ANTHROPIC_AUTH_TOKEN:?}" +: "${SKILLSBENCH_TASKS_DIR:?}" + +# Snapshot config +{ + echo + echo "===== .env (redacted) =====" + sed 's/\(TOKEN\|KEY\)=.*/\1=/' "$PROJECT_ROOT/.env" + echo + echo "===== git commit =====" + ( cd "$PROJECT_ROOT" && git log -1 --format='commit %H%n%s' 2>&1 || echo "(not a git repo)" ) + echo + echo "===== podman info | head -30 =====" + podman info 2>&1 | head -30 +} >> "$ENV_SNAP" + +# --- Phase 3: run bench --- +banner "Phase 3: bench eval run --include $TASK" +JOBS="$OUT_DIR/bench_jobs" +rm -rf "$JOBS" + +set +e +stdbuf -oL -eL bench eval run \ + --tasks-dir "$SKILLSBENCH_TASKS_DIR" \ + --include "$TASK" \ + --agent claude-agent-acp --model claude-opus-4-6 \ + --sandbox docker --sandbox-user '' \ + --skill-mode with-skill \ + --skills-dir "$PROJECT_ROOT/.capevolve/project/seed_capability" \ + --jobs-dir "$JOBS" \ + --agent-env "ANTHROPIC_BASE_URL=$ANTHROPIC_BASE_URL" \ + --agent-env "ANTHROPIC_AUTH_TOKEN=$ANTHROPIC_AUTH_TOKEN" 2>&1 | tee "$BENCH_LOG" +RC="${PIPESTATUS[0]}" +set -e + +# --- Phase 4: summarize + mark outcome --- +banner "Phase 4: results" +echo "End: $(date -Iseconds)" +echo "Exit: $RC" + +# Newest run subdir (timestamped) +RESULT_JSON=$(find "$JOBS" -name result.json 2>/dev/null | head -1) +if [[ -z "$RESULT_JSON" ]]; then + echo "No result.json produced — bench never got to a rollout." + echo "ERROR" > "$OUT_DIR/OUTCOME" + exit "${RC:-1}" +fi + +echo +echo "===== result summary =====" +python3 -c " +import json +d = json.load(open('$RESULT_JSON')) +ar = d.get('agent_result',{}) or {} +err = d.get('error') or 'None' +r = d.get('rewards') or 'None' +print(f'rollout: {d.get(\"rollout_name\")}') +print(f'error: {str(err)[:200]}') +print(f'error_category:{d.get(\"error_category\")}') +print(f'rewards: {r}') +print(f'tool_calls: {d.get(\"n_tool_calls\")}') +print(f'out_tokens: {ar.get(\"n_output_tokens\")}') +print(f'in_tokens: {ar.get(\"n_input_tokens\")}') +print(f'cost_usd: {ar.get(\"cost_usd\")}') +print(f'timing: {d.get(\"timing\")}')" + +# Decide outcome +OUTCOME=$(python3 -c " +import json +d = json.load(open('$RESULT_JSON')) +err = d.get('error') +rewards = d.get('rewards') or {} +reward = rewards.get('reward') if isinstance(rewards, dict) else None +if err: + print('ERROR') +elif reward == 1.0: + print('PASS') +elif reward == 0.0: + print('FAIL') +else: + print('UNKNOWN') +") +echo "$OUTCOME" > "$OUT_DIR/OUTCOME" +echo +echo "OUTCOME: $OUTCOME" +echo "Results: $OUT_DIR" + +# Exit 0 on PASS/FAIL (plumbing works), non-zero on ERROR +case "$OUTCOME" in + PASS|FAIL) exit 0 ;; + *) exit 1 ;; +esac diff --git a/scripts/ccc/setup_podman.sh b/scripts/ccc/setup_podman.sh new file mode 100755 index 00000000..2183eecb --- /dev/null +++ b/scripts/ccc/setup_podman.sh @@ -0,0 +1,238 @@ +#!/bin/bash +# +# Podman setup for CCC. +# +# Usage: +# source /scripts/ccc/setup_podman.sh +# +# Or run once to create ~/.config/containers/storage.conf, then just export +# XDG_RUNTIME_DIR yourself in later shells. +# +# What this does: +# - Writes ~/.config/containers/storage.conf pointing runroot/graphroot at +# host-local /tmp paths (so different LSF hosts don't corrupt each other). +# - Exports XDG_RUNTIME_DIR to a writable /tmp path (the default +# /run/user/$UID isn't writable for us on CCC). +# - Creates the directories if missing. +# +# Notes: +# - Storage lives on /tmp -> host-local. Image pulls don't survive host +# changes. If image pulls become a bottleneck, switch STORAGE_BASE to a +# GPFS path suffixed with $(hostname) (see below). +# - You may still see a subuid/subgid warning on first run. Simple +# containers work; images that need user namespaces require CCC admins +# to add you to /etc/subuid and /etc/subgid. Ping @ccc_admins in that +# case. +# - Workaround when admin help is unavailable: we set +# ignore_chown_errors="true" under [storage.options.overlay]. Podman +# then skips lchown calls it can't perform during image unpack; files +# that "should" be owned by UID 42 (e.g. /etc/gshadow) land as root +# instead. Fine for most tasks; tasks that verify file ownership inside +# the container will still break. + +_uid="$(id -u)" + +# Host-local storage. Safe for concurrent podman on the same host (podman +# serializes internally), never shared across hosts. +STORAGE_BASE="/tmp/podman-${_uid}" +RUN_BASE="/tmp/podman-run-${_uid}" + +# Alternative for GPFS-backed image cache (uncomment and comment the two +# lines above). Pays a per-host directory but images persist per host. +# STORAGE_BASE="/dccstor///.podman-$(hostname)" +# RUN_BASE="/tmp/podman-run-${_uid}" + +mkdir -p "$STORAGE_BASE" "$RUN_BASE" +chmod 700 "$STORAGE_BASE" "$RUN_BASE" + +# Write storage.conf if missing OR if it doesn't match the paths we want. +CONF_DIR="$HOME/.config/containers" +CONF_FILE="$CONF_DIR/storage.conf" +mkdir -p "$CONF_DIR" + +_desired=$(cat </dev/null 2>&1; then + printf '%s\n' "$_desired" > "$CONF_FILE" + echo "[setup_podman] wrote $CONF_FILE" +fi + +export XDG_RUNTIME_DIR="$RUN_BASE" + +# Shim `docker` in userspace: /usr/bin/docker (podman-docker shim) prints +# "Emulate Docker CLI using podman. Create /etc/containers/nodocker to +# quiet msg." to STDOUT on every invocation. Tools that capture the stdout +# of `docker`-run commands (e.g. bench probing the container's `pwd`) end +# up with that string embedded in their result, which then gets passed as +# an argument to subsequent execs and breaks them. Our shim goes to podman +# directly with no chatty prefix. We can't `touch /etc/containers/nodocker` +# on CCC without root. +_docker_shim="$HOME/.local/bin/docker" +if [ ! -x "$_docker_shim" ]; then + mkdir -p "$HOME/.local/bin" + cat > "$_docker_shim" <<'DOCKER_SHIM' +#!/bin/sh +exec podman "$@" +DOCKER_SHIM + chmod +x "$_docker_shim" +fi +# Always PREPEND: if $HOME/.local/bin was already in PATH but after /usr/bin, +# our shim wouldn't win the docker lookup. Strip any prior occurrence, then +# prepend so our shim always resolves first. +_new_path=":$PATH:" +_new_path="${_new_path//:$HOME\/.local\/bin:/:}" +_new_path="${_new_path#:}"; _new_path="${_new_path%:}" +export PATH="$HOME/.local/bin:$_new_path" +unset _new_path + +# Private DBus session bus. Aardvark-dns (podman's built-in DNS resolver, +# invoked by netavark when a container joins a bridge network) tries to +# talk to a session bus. On compute nodes there's no systemd user +# session, so `/run/user/$UID/bus` doesn't exist and container startup +# fails with "aardvark-dns failed to start: Failed to connect to bus". +# Fix: start our own dbus-daemon and export DBUS_SESSION_BUS_ADDRESS. +_dbus_sock="$RUN_BASE/dbus.sock" +_dbus_pidfile="$RUN_BASE/dbus.pid" +_dbus_alive() { + [ -S "$_dbus_sock" ] && [ -f "$_dbus_pidfile" ] && \ + kill -0 "$(cat "$_dbus_pidfile")" 2>/dev/null +} +if ! _dbus_alive; then + pkill -u "$(id -u)" -f "dbus-daemon.*$_dbus_sock" 2>/dev/null || true + sleep 0.2 + rm -f "$_dbus_sock" + if command -v dbus-daemon >/dev/null 2>&1; then + dbus-daemon --session --address="unix:path=$_dbus_sock" --fork \ + --print-pid > "$_dbus_pidfile" 2>"$RUN_BASE/dbus.log" + for _i in 1 2 3 4 5; do + [ -S "$_dbus_sock" ] && break + sleep 0.3 + done + else + echo "[setup_podman] WARN: dbus-daemon not found; container DNS may fail" + fi +fi +export DBUS_SESSION_BUS_ADDRESS="unix:path=$_dbus_sock" + +# Rootless podman socket for `docker compose` (and any other Docker API +# client) to talk to. On login nodes systemd user sessions manage this via +# `podman.socket`; on compute nodes there's no systemd user session, so we +# start `podman system service` ourselves as a background process. +# +# Idempotence: SQLite backend can't handle multiple podman services +# writing to the same graphroot — a socket file left behind by a +# previously-killed service passes `-S` but the service is dead. Check +# BOTH socket exists AND a service process is alive; kill all instances +# and start exactly one otherwise. +_sock="$RUN_BASE/podman.sock" +_pidfile="$RUN_BASE/podman-service.pid" +_service_alive() { + [ -S "$_sock" ] && [ -f "$_pidfile" ] && kill -0 "$(cat "$_pidfile")" 2>/dev/null +} +if ! _service_alive; then + # Reap any zombie services that leaked from prior sessions before we + # start a new one, otherwise a stale one holds the DB and the new + # service's writes fail with "readonly database". + pkill -u "$(id -u)" -f "^podman system service .*$_sock" 2>/dev/null || true + sleep 0.2 + rm -f "$_sock" + nohup podman system service --time=0 "unix://$_sock" \ + >"$RUN_BASE/podman-service.log" 2>&1 & + echo $! > "$_pidfile" + disown 2>/dev/null || true + for _i in 1 2 3 4 5; do + [ -S "$_sock" ] && break + sleep 0.3 + done +fi +export DOCKER_HOST="unix://$_sock" + +# Second subuid workaround: apt inside the container tries to drop +# privileges to the `_apt` user (UID 42). With no /etc/subuid entry, that +# setuid fails and `apt-get install` dies. Fix: shadow the ubuntu:24.04 +# base image locally with an /etc/apt config that tells apt to stay as +# root. Any downstream FROM ubuntu:24.04 inherits this and Just Works. +_patched_marker="$STORAGE_BASE/.patched-ubuntu24-v5" +if [ ! -f "$_patched_marker" ]; then + _patch_dockerfile=$(mktemp /tmp/patched-ubuntu.XXXXXX.Dockerfile) + cat > "$_patch_dockerfile" <<'DOCKERFILE' +FROM docker.io/library/ubuntu:24.04 +# 1) apt itself tries to drop privileges to `_apt` (UID 42); without a +# subuid mapping we can only be UID 0. Tell apt to stay as root. +RUN echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf.d/00-rootless +# 2) Skip Recommends so we don't drag in build-essential, libgd3, +# libc-devtools, fontconfig-config — their postinst scripts do +# chown/adduser to non-root UIDs and fail without subuid. +RUN echo 'APT::Install-Recommends "false";' >> /etc/apt/apt.conf.d/00-rootless +# 3) The MOST COMMON downstream failure: package postinst scripts do +# `chown fontconfig:root /var/cache/fontconfig` or similar, which +# fails with "Invalid argument" in our single-UID user namespace. +# Wrap chown/chgrp to swallow ownership errors — the file itself is +# created fine, only the ownership call fails. Same for useradd's +# setgid step (via wrapping the binary or by pre-creating users). We +# wrap chown+chgrp; useradd/groupadd typically also fail-open with +# warnings that dpkg accepts. +RUN set -e; \ + for bin in /usr/bin/chown /usr/bin/chgrp \ + /usr/sbin/useradd /usr/sbin/groupadd \ + /usr/sbin/usermod /usr/sbin/groupmod \ + /usr/sbin/adduser /usr/sbin/addgroup \ + /usr/sbin/dpkg-statoverride /usr/bin/dpkg-statoverride; do \ + [ -x "$bin" ] || continue; \ + mv "$bin" "${bin}.real"; \ + printf '%s\n' '#!/bin/sh' "${bin}.real \"\$@\" 2>/dev/null || :" > "$bin"; \ + chmod +x "$bin"; \ + done +# 4) Pre-install what most SkillsBench Dockerfiles ask for. Downstream +# `apt-get install python3 python3-pip curl poppler-utils` finds the +# common ones present and only fetches deltas. build-essential is +# heavy (~500 MB) but multiple tasks need it; better to pay once here +# than on every rollout's image build. +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + python3 python3-pip curl \ + poppler-utils \ + build-essential \ + && rm -rf /var/lib/apt/lists/* +DOCKERFILE + # Force the tag off any pre-existing local image before rebuilding. + podman rmi -f docker.io/library/ubuntu:24.04 >/dev/null 2>&1 || true + if podman build \ + -t docker.io/library/ubuntu:24.04 \ + -f "$_patch_dockerfile" \ + "$(dirname "$_patch_dockerfile")" >"$RUN_BASE/patched-ubuntu-build.log" 2>&1; then + touch "$_patched_marker" + echo "[setup_podman] patched ubuntu:24.04 for apt-in-rootless (log: $RUN_BASE/patched-ubuntu-build.log)" + else + echo "[setup_podman] WARN: could not pre-patch ubuntu:24.04; container builds may fail on apt (log: $RUN_BASE/patched-ubuntu-build.log)" + fi + rm -f "$_patch_dockerfile" +fi + +# Sanity summary (previously interactive-only; now always emitted so batch +# runs get a non-empty setup.log for post-mortem). +echo "[setup_podman] XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR" +echo "[setup_podman] graphroot=$STORAGE_BASE" +echo "[setup_podman] DOCKER_HOST=$DOCKER_HOST" +echo "[setup_podman] DBUS_SESSION_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS" +echo "[setup_podman] PATH first entry: $(echo "$PATH" | cut -d: -f1)" +echo "[setup_podman] try: podman info | head -30" + +unset _uid _desired _sock _i _patched_marker _patch_dockerfile _pidfile +unset _dbus_sock _dbus_pidfile _docker_shim +unset -f _service_alive _dbus_alive 2>/dev/null || true diff --git a/scripts/ccc/submit_ccc_experiment.sh b/scripts/ccc/submit_ccc_experiment.sh new file mode 100755 index 00000000..8adb6f41 --- /dev/null +++ b/scripts/ccc/submit_ccc_experiment.sh @@ -0,0 +1,147 @@ +#!/bin/bash +# +# Submit a cap-evolve experiment to LSF (CCC batch). +# +# Wraps `bsub` around scripts/ccc/run_ccc_experiment.sh with sensible +# defaults for a full baseline or 7-iter run. Modeled on +# skillberry-skill-maker's run_tasks_on_dedicated_machines.sh. +# +# Usage: +# bash scripts/ccc/submit_ccc_experiment.sh \ +# --suite-id baseline_v1 --max-iterations 0 +# +# # Full 7-iter run +# bash scripts/ccc/submit_ccc_experiment.sh \ +# --suite-id iter7_opus_v1 --max-iterations 7 \ +# --queue x86_1h --memory 64G --walltime 6:00 +# +# # Dry-run — print the bsub command without submitting +# bash scripts/ccc/submit_ccc_experiment.sh \ +# --suite-id test --max-iterations 0 --dry-run +# +# Options: +# --suite-id ID [REQUIRED] logical grouping (e.g. "baseline_v1") +# --max-iterations N 0 for baseline (default), 7 for full evolve +# --spec PATH path to capevolve.yaml (default: .capevolve/project/capevolve.yaml) +# --project PATH path to .capevolve/project (default: .capevolve/project) +# --queue Q LSF queue (default: x86_6h) +# --memory MEM memory per job (default: 64G) +# --walltime H:MM wall-clock limit (default: 6:00 for iter=0, 12:00 for iter>0) +# --cpus N CPU slots (default: 4) +# --extra-args "..." verbatim flags passed to cap-evolve run +# --dry-run print the bsub command, don't submit +# +# The run-id is auto-derived from LSF's job ID once submitted (so results +# land under results///). + +set -euo pipefail + +# Defaults +SUITE_ID="" +MAX_ITERATIONS="0" +SPEC=".capevolve/project/capevolve.yaml" +PROJECT_DIR=".capevolve/project" +QUEUE="x86_6h" +MEMORY="64G" +WALLTIME="" # auto by MAX_ITERATIONS below +CPUS="4" +EXTRA_ARGS="" +DRY_RUN=false +CCC_LOGS="${CCC_LOGS:-$HOME/ccc_logs}" + +while [[ $# -gt 0 ]]; do + case "$1" in + --suite-id) SUITE_ID="$2"; shift 2 ;; + --max-iterations) MAX_ITERATIONS="$2"; shift 2 ;; + --spec) SPEC="$2"; shift 2 ;; + --project) PROJECT_DIR="$2"; shift 2 ;; + --queue) QUEUE="$2"; shift 2 ;; + --memory) MEMORY="$2"; shift 2 ;; + --walltime) WALLTIME="$2"; shift 2 ;; + --cpus) CPUS="$2"; shift 2 ;; + --extra-args) EXTRA_ARGS="$2"; shift 2 ;; + --dry-run) DRY_RUN=true; shift ;; + -h|--help) sed -n '2,40p' "$0"; exit 2 ;; + *) echo "Unknown option: $1" >&2; exit 2 ;; + esac +done + +if [[ -z "$SUITE_ID" ]]; then + echo "ERROR: --suite-id is required" >&2 + exit 2 +fi + +if [[ -z "$WALLTIME" ]]; then + # Baseline is ~1h. Full 7-iter is 4-6h; give margin. + if [[ "$MAX_ITERATIONS" == "0" ]]; then + WALLTIME="2:00" + else + WALLTIME="8:00" + fi +fi + +# Resolve project root: this script lives at $PROJECT_ROOT/scripts/ccc/*.sh +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + +INNER="$PROJECT_ROOT/scripts/ccc/run_ccc_experiment.sh" +[[ -x "$INNER" ]] || { echo "ERROR: inner script not executable: $INNER" >&2; exit 2; } + +mkdir -p "$CCC_LOGS" + +JOB_NAME="capevolve_${SUITE_ID}_iter${MAX_ITERATIONS}" + +# Build bsub command. -oo/-eo to keep separate stdout/stderr per LSF job id. +BSUB_CMD=( + bsub + -q "$QUEUE" + -M "$MEMORY" + -n "$CPUS" + -W "$WALLTIME" + -J "$JOB_NAME" + -oo "${CCC_LOGS}/%J.stdout" + -eo "${CCC_LOGS}/%J.stderr" + bash "$INNER" + --suite-id "$SUITE_ID" + --max-iterations "$MAX_ITERATIONS" + --spec "$SPEC" + --project "$PROJECT_DIR" +) +if [[ -n "$EXTRA_ARGS" ]]; then + BSUB_CMD+=(--extra-args "$EXTRA_ARGS") +fi + +printf 'PROJECT_ROOT: %s\n' "$PROJECT_ROOT" +printf 'JOB: %s\n' "$JOB_NAME" +printf 'QUEUE: %s\n' "$QUEUE" +printf 'MEMORY: %s\n' "$MEMORY" +printf 'CPUs: %s\n' "$CPUS" +printf 'WALLTIME: %s\n' "$WALLTIME" +printf 'LOG DIR: %s\n' "$CCC_LOGS" +printf 'INNER CMD: %s\n' "${BSUB_CMD[*]}" + +if [[ "$DRY_RUN" == true ]]; then + echo + echo "(dry-run: not submitting)" + exit 0 +fi + +echo +"${BSUB_CMD[@]}" +echo +cat < + tail -f ${CCC_LOGS}/.stdout + +Results will land at: + $PROJECT_ROOT/results/$SUITE_ID// + + results/${SUITE_ID}//setup.log # setup_podman.sh output + results/${SUITE_ID}//cap-evolve.log # cap-evolve stdout+stderr + results/${SUITE_ID}//env_snapshot.txt # .env + capevolve.yaml + git commit + podman info + results/${SUITE_ID}//run/ # cap-evolve's run dir (symlink) + +EOF