Concurrent rollouts race on os.environ: pi-acp launcher and openclaw shim mutate process env per rollout

[bingran-you]

Summary

Two agent-launch helpers mutate os.environ directly during per-rollout setup. In a multi-rollout (concurrent) eval — the default usage pattern at any meaningful scale — these writes race across rollouts running in the same Python process. The last writer wins; earlier rollouts can be misconfigured silently.

Concrete file:line

• src/benchflow/agents/pi_acp_launcher.py:113-117 — writes provider URL/key env vars to os.environ.
• src/benchflow/agents/openclaw_acp_shim.py:138, :358 — same pattern.

These run per-rollout (each Rollout.run), but os.environ is process-global. Two rollouts launched concurrently with different agents/models can interleave their env mutations.

Concrete impact

# Rollout A (model=gemini-2.5-flash) sets GEMINI_BASE_URL=https://a/
# Rollout B (model=gemini-2.5-pro)   sets GEMINI_BASE_URL=https://b/
# Rollout A's subsequent agent invocation now reads B's URL.

For --concurrency > 1 runs the bug surfaces probabilistically. Symptoms:

• Wrong model called.
• Wrong provider URL.
• Cross-test contamination in CI.

Severity

P2.

• Single-task evals are unaffected.
• Multi-task evals on the same process trigger the race.
• The default --concurrency is documented as 4 (varies by sandbox) — the race surface is non-trivial.

Proposed fix

1. Stop mutating os.environ. Build a per-rollout env: dict[str, str] and pass it to the child process via subprocess/asyncio.create_subprocess_* env= kwarg. The per-rollout env never touches the process env.

2. If the launcher truly needs the env to be visible to nested calls (e.g. an SDK reads os.environ lazily), scope the mutation with contextvars.ContextVar or a with patch.dict(os.environ, ...) context manager that restores on exit — and serialize concurrent rollouts that hit this code path.

3. Add a regression test that runs two rollouts concurrently with conflicting env values and asserts they don't cross-contaminate.

Related

Companion to #418 (closed: "eval prunes Docker globally"). Both are "per-rollout helper mutates process-global state" antipattern — the global-Docker-prune fix should set the pattern, and these two launchers should follow it.

Found via

v0.5 e2e validation arch-audit at 5a7a326.

[bingran-you]

Automation re-validation (2026-06-16): premise appears invalid on the current architecture — recommend re-check/close.

The pi-acp launcher and openclaw shim are materialized to disk as standalone scripts (registry.py:375/402) and executed as separate subprocess entrypoints (if __name__ == "__main__" → os.execv). There is no in-process import of their functions anywhere in the tree (grep = 0 hits). So each rollout's launcher mutates only its own subprocess environment — the same-Python-process os.environ race this issue describes cannot occur as written.

@bingran-you you filed this as reproduced on v0.5-integration — if the concurrent-race symptom is no longer observed on main, this is closable (and reproduced should be dropped). Leaving open pending your confirmation since the original repro was yours.

[bingran-you]

Automation triage update (2026-06-18): added this back to the benchflow-ai/tasks project board as Todo, but I did not find current evidence that the original os.environ race is still reproducible on the current rollout architecture.

Keeping the issue open and moving it from status:ready to status:blocked + question: before making repo code changes here, we need a fresh concurrent-rollout repro or a precise current code path showing per-rollout environment mutation still leaks across runs.

[bingran-you]

Automation triage (2026-06-19): no BenchFlow code change started today because the current repro needs to be refreshed.

The code still contains launcher/shim env mutation patterns, but the scanned paths are standalone launcher processes rather than obviously shared in-process rollout setup. The existing labels question + status:blocked are appropriate. Please provide a current-main repro showing cross-rollout contamination in one Python process, or confirm the desired policy is to eliminate all process-env mutation in launcher scripts even without a live race repro.