feat(agent-runtime): OpenAI Codex as the third agent runtime + resolve E2E findings (#1187)

[AndriiPasternak31]

Summary

Lands the #1187 "harness == runtime" work — OpenAI Codex joins Claude Code and Gemini as a pluggable AgentRuntime inside the agent container — and resolves the two blockers the live E2E surfaced that kept a Codex agent from acting.

A Trinity harness IS an AgentRuntime. AGENT_RUNTIME (from template.yaml runtime:, also a trinity.agent-runtime label) selects one; runtime_adapter.get_runtime() validates against KNOWN_RUNTIMES and raises on an unknown value.

What's in it

Core runtime (codex_runtime.py, built on the per-runtime primitives — not a shared helper, so it never inherits Gemini's blanket kill_cgroup_orphans()):

• codex exec --json JSONL parse (thread.started / turn.completed.usage / item.completed); reasoning_output_tokens ⊂ output_tokens (no double-count); -o output file authoritative (read-then-delete in finally); estimated cost (CODEX_PRICING).
• chat continuity via codex exec resume <thread_id>; concurrency-safe _drain_bounded orphan cleanup; error→HTTP auth 503 / rate 429 / runtime-unavailable 500-not-503 / pipe-drop 502.
• RuntimeCapabilities dataclass + per-runtime capabilities(); Codex skips Claude-subscription assign (is_claude_runtime); OPENAI_API_KEY read from .env; CODEX_HOME under $TMPDIR (gitignored). Session tab gated off via RUNTIMES_WITHOUT_SESSION_TAB_RESUME; frontend RuntimeBadge/terminal/default-model; test-codex template; @openai/codex@0.139.0 pinned.

E2E findings resolved (the four fixes):

1. Resume arg-order — exec-level flags (--json/-C/--sandbox/-o) must precede the resume sub-subcommand (codex 0.139.0 rejects them after → exit 2 → every turn-2+ failed). Fixed + arg-order guard test.
2. F-TOOLS (sandbox) — normal mode → --sandbox danger-full-access. Codex's own bubblewrap sandbox can't create a user namespace inside the hardened agent container (bwrap: No permissions to create a new namespace), which blocked every shell tool. Dropping it leaves the Trinity container as the sole boundary. Read-only keeps --sandbox read-only.
3. F-MCP (runtime-aware prompt) — PLATFORM_INSTRUCTIONS documented MCP tools with Claude's mcp__trinity__<tool> prefix; Codex auto-discovers tools by bare name and answered unknown MCP server. get_platform_system_prompt(runtime=…) / compose_system_prompt(runtime=…) strip the prefix + add a Codex orientation note for runtime="codex", threaded from routers/chat.py + services/task_execution_service.py via the trinity.agent-runtime label (docker_service.get_agent_runtime, resolved lazily + guarded so a re-import under a stubbed docker_service in unit tests can't break dispatch). Claude/Gemini/unknown unchanged.
4. Test-isolation — test_validate_runtime.py hardened against the sibling sys.modules Mock leak.

Housekeeping: cap bcrypt<5 in tests/requirements-test.txt (bcrypt 5.0.0 removed the __about__ shim passlib 1.7.4 reads at import → breaks every test at conftest DB-init in a fresh venv).

🔒 Security note

Disabling Codex's internal bubblewrap sandbox (danger-full-access) does not weaken container hardening. The Trinity agent container is already the security boundary — cap_drop: ALL, AppArmor, no-new-privileges, network isolation — and runs with no userns grant and unchanged caps. This is exactly the posture under which Claude Code and Gemini already run (no internal sandbox). The change drops a redundant inner sandbox that simply cannot function inside the hardened container.

Test plan

• Unit: full tests/unit green — 2218 passed, 24 skipped, 0 fail (incl. new sandbox-resolution, runtime-aware-prompt, and resume arg-order tests). Caught + fixed a real order-dependency I'd briefly introduced (top-level docker_service import → lazy+guarded), re-verified under adversarial ordering.

• Live E2E on a real Codex agent booted from the rebuilt base image (key injected via CRED-002):

  Check	Result
  F-TOOLS	file created + read back (HELLO_CODEX_E2E); log sandbox=danger-full-access; tools ran (was bwrap fail)
  F-MCP	list_agents via Trinity MCP returned the agent (was unknown MCP server)
  turn-1	cost ≈ $0.046, session id
  turn-2 resume	recalled prior-turn context with no tools; log resume=True
  read-only	log sandbox=read-only
  credential leak	sk-proj ×0 in agent logs

  danger-full-access ran tools non-interactively (no approval prompt → no --dangerously-bypass-approvals-and-sandbox needed).

• verify-local --skip-agent (the agent side was validated more thoroughly by the live E2E above; the dev stack held the global agent network).

Docs

architecture.md, feature-flows.md + feature-flows/codex-runtime.md, harness-authoring-guide.md, requirements.md, agent guide.

Implements #1187.

ℹ️ See the follow-up comment on read-only enforcement for Codex — an open design decision for the team (Codex has no PreToolUse hook).

🤖 Generated with Claude Code

[AndriiPasternak31]

🟡 Open design decision: read-only enforcement for Codex

This PR ships read-only for Codex as --sandbox read-only (verified: a read-only turn logs [Codex] exec sandbox=read-only). I want to flag the underlying tradeoff for a team decision, because Codex's read-only story is weaker than Claude's and I don't want it to slip in unexamined.

The gap: Claude enforces read-only via a PreToolUse hook (read-only-guard.py) that inspects every tool call against blocked/allowed patterns. Codex has no PreToolUse hook, so we can't replicate that per-pattern enforcement. Its only native read-only mechanism is --sandbox read-only — but that sandbox is bubblewrap, the same mechanism we just had to drop for normal mode because it can't create a user namespace inside the hardened container (bwrap: No permissions to create a new namespace).

So there's a real question of whether --sandbox read-only actually enforces anything for Codex inside our container, or whether bwrap fails the same way and the agent silently has no sandbox. (In this PR's E2E the read-only turn was a no-tool prompt, so it proves the flag is wired, not that writes are blocked — that needs a dedicated write-attempt test.)

Options for the team:

• (A) fail-closed — keep --sandbox read-only; if bwrap can't init, tools error → no writes possible → agent is effectively chat-only in read-only mode. Safe; this is what the PR ships as the interim.
• (B) advisory — danger-full-access + a strong read-only prompt directive. Tools work, but write-protection is soft/unenforced (prompt-only).
• (C) unsupported — block/warn read-only for Codex in the MVP and revisit when Codex exposes a tool-gating hook.

The PR ships (A) as the safe interim. Requesting a decision before this hardens into the contract — and either way we should add a write-attempt test that asserts a blocked write actually fails under read-only.

[github-actions]

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

[vybe]

✅ Validated & approved (/validate-pr) — clean, no defects. Recommend a /cso --diff security pass before merge (see below).

P1 infrastructure feature (Closes #1187). 42 files, +4423/-41 — large, but the bulk is the new runtime + 12 test files + thorough docs; scope is coherent.

Validation — all green:

• Base → dev ✓ · 7 conventional commits ✓ · full CI matrix green (build, verify-non-root, schema-parity, prod-image-smoke, pytest ×6 seeds, CodeQL, regression-diff)
• Docs exceptional: architecture + requirements + feature-flows index + new codex-runtime.md + harness-authoring-guide + TRINITY_COMPATIBLE_AGENT_GUIDE. New capability → requirements + feature-flow both present ✓
• Security scan clean: no real secrets (the trinity_mcp_SUPERSECRETVALUE hit is a test fixture in test_codex_mcp_config.py; the sk-* strings are test fixtures); MCP token referenced via bearer_token_env_var (never persisted); OPENAI_API_KEY read from .env/process env, not exported into the agent-server process; CODEX_HOME under $TMPDIR + gitignored.
• Invariant #5 (agent-server + backend both updated) ✓; #13 (no MCP-server change needed — Codex consumes the existing Trinity MCP via config.toml) ✓; no Dockerfile COPY gap (agent_server/ copied wholesale, so codex_runtime.py ships) ✓; no new backend env vars.
• codex_runtime.py correctly built on per-runtime primitives (never inherits Gemini's blanket kill_cgroup_orphans()); resume arg-order fix + guard test; runtime-aware prompt (strips mcp__trinity__ for Codex); cost/token parse avoids reasoning double-count.

Sandbox posture change — reviewed, sound: normal mode → --sandbox danger-full-access drops Codex's inner bubblewrap (which can't create a userns in the hardened container → blocks every shell tool). Claude/Gemini already run with no inner sandbox, so this is parity, not a regression — the Trinity container (non-root, CAP_DROP, network isolation) remains the boundary. Read-only mode correctly keeps --sandbox read-only via ~/.trinity/read-only-config.json (fail-open consistent with the Claude reference hook).

⚠️ Recommended before merge (P1 methodology gate): run /cso --diff for a deep security pass focused on the danger-full-access decision, the Codex subprocess env/credential flow, and the credential sanitizer over Codex output — /validate-pr + CodeQL are green, but the full P1 pipeline (/review + /validate-pr + /cso) prescribes it for a sandbox-posture change.

Merge note: this shares task_execution_service.py + docs/memory/architecture.md + docker/base-image/Dockerfile with #1195/#1196 — land it last (it pays the rebase). Base-image rebuild required (@openai/codex@0.139.0).