Skip to content

CODEX_AGENT_GUIDE.md

Latest commit

 

History

History
289 lines (213 loc) · 11.8 KB

CODEX_AGENT_GUIDE.md

File metadata and controls

289 lines (213 loc) · 11.8 KB
Raw

Codex Agent Implementation

Overview

A new CodexAgent has been added to coder_eval that integrates OpenAI's Codex SDK. The implementation mirrors the structure of ClaudeCodeAgent and provides seamless integration with the evaluation framework.

Setup

1. Install the Codex SDK

Install coder-eval with the codex extra:

pip install 'coder-eval[codex]'

This installs:

  • openai-codex - The official Codex Python SDK (from PyPI)
  • openai-codex-cli-bin - Platform-specific Codex CLI binaries (pulled in transitively)

2. Authentication

Codex requires authentication. Options:

# Option 1: API Key (direct)
await codex_client.login_api_key("your-api-key")

# Option 2: ChatGPT (interactive)
await codex_client.login_chatgpt()

# Option 3: Device Code Flow
await codex_client.login_chatgpt_device_code()

CodexAgent.start() calls login_api_key automatically when CODEX_API_KEY is present in the environment. Without a key it falls back to any existing ChatGPT login. (Only CODEX_API_KEY is read — not OPENAI_API_KEY/AZURE_OPENAI_API_KEY; point CODEX_API_KEY at whichever endpoint's key you use.)

Endpoint routing

Env var Purpose
CODEX_API_KEY Auth key/token for the selected endpoint (required for headless runs).
CODEX_BASE_URL Route to a custom endpoint. Unset → standard OpenAI platform (api.openai.com). Set → custom provider (gateway or Azure).
CODEX_MODEL Fallback model when agent.model is unset. On Azure this is the deployment name.
CODEX_API_VERSION Azure only: the required api-version query param. Leave unset for OpenAI/gateways.

Standard OpenAI: leave CODEX_BASE_URL unset, set CODEX_API_KEY to an OpenAI sk-… key and CODEX_MODEL to a Codex/Responses-capable model.

Azure OpenAI:

CODEX_BASE_URL=https://<your-resource>.openai.azure.com/openai
CODEX_API_VERSION=2025-04-01-preview   # required by Azure
CODEX_MODEL=<your-deployment-name>     # deployment, not the base model id
CODEX_API_KEY=<azure-openai-key>

This registers a custom Codex model provider (base_url + env_key=CODEX_API_KEY + query_params={api-version} + wire_api=responses). The Codex CLI only supports the Responses wire API (it rejects wire_api=chat as "no longer supported"), so the protocol is fixed. If your Azure deployment requires the key in an api-key header rather than Authorization: Bearer, that needs an additional provider http_headers/env_http_headers entry — open an issue if you hit that.

Usage

Command Line

Run a task with Codex agent:

coder-eval run tasks/agents/codex_hello_world.yaml --type codex

Or override agent type for all tasks in an experiment:

coder-eval run experiments/example.yaml --type codex

Task Definition (YAML)

Specify Codex in task YAML:

agent:
  type: codex
  permission_mode: acceptEdits
  allowed_tools:
    - Bash
    - Read
    - Write
  disallowed_tools:
    - Edit
  plugins:
    - type: local
      path: "$PLUGIN_PATH"

success_criteria:
  - type: file_exists
    path: "src/solution.py"
    description: "Solution file must exist"

Valid permission_mode values:

  • default - Standard access, requires approval on failure
  • acceptEdits - Automatically accept file edits, no filesystem restrictions
  • plan - Read-only sandbox, approval required for any changes
  • bypassPermissions - Full access, no approvals needed

Skills (SKILL.md)

CodexAgent supports SKILL.md files following the Agent Skills open standard. Skills are discovered from:

  1. config.plugins - Local plugins with type: local and path pointing to a skills directory
  2. plugin_tools_dir parameter - Runtime plugin directory passed to start()

Skills are symlinked (or copied) to .agents/skills/ where the Codex CLI auto-discovers them. Environment variables in plugin paths ($VAR, ${VAR}) are expanded at runtime.

Example with environment variable:

agent:
  type: codex
  plugins:
    - type: local
      path: "$SKILLS_PLUGIN_PATH"

Set environment variable:

export SKILLS_PLUGIN_PATH=~/uipath/uipath-claude-plugins/plugins/uipath-coded-agents
coder-eval run tasks/my_task.yaml

Architecture

Class Hierarchy

Agent (ABC)
└── CodexAgent
    ├── Codex SDK Client (openai_codex.Codex)
    ├── Thread Management (thread_start, turn.stream)
    └── Streaming telemetry (commands, token usage, agent text)

Key Methods

  • start(working_directory) - Initialize Codex client and set working directory
  • communicate(user_input, timeout, stream_callback) - Execute one turn with Codex
  • stop() - Clean up resources
  • get_state() - Return current agent state
  • discard_pending_turn() - Rollback on failure

TurnRecord Format

Each turn returns a TurnRecord with:

  • iteration - Turn number
  • user_input - The prompt sent
  • agent_output - assembled from the streamed agentMessage deltas
  • commands - CommandTelemetry for each shell command (Bash) and apply_patch file change (Write)
  • timestamp - When the turn completed
  • duration_seconds - Wall-clock execution time
  • token_usage - input/output/cache-read token counts (from the SDK token-usage stream)
  • model_used - the pinned agent.model, when set

Implementation Details

Timeout Handling

The agent uses a ThreadedWatchdog to enforce wall-clock timeouts. If a turn exceeds the deadline, a TurnTimeoutError is raised with a partial TurnRecord preserved in pending_turn.

Error Recovery

On failure, the agent:

  1. Sets pending_turn to a crashed=True TurnRecord with captured telemetry
  2. Raises AgentCrashError or TurnTimeoutError
  3. The orchestrator reads pending_turn and calls discard_pending_turn() to roll back state

Permission and Tool Mapping

The agent maps permission_mode to the Codex SDK's Sandbox. The approval mode is uniformly deny_all for every mode — the trust boundary is the sandbox, which does vary by mode:

permission_mode sandbox approval_mode
bypassPermissions full-access deny_all
acceptEdits workspace-write deny_all
default workspace-write deny_all
plan read-only deny_all

deny_all means run autonomously, never prompt, no server-side reviewer: in-sandbox operations execute directly and only escalations beyond the sandbox are refused. coder_eval uses it for every mode because the alternative (auto_review) adds a server-side reviewer that can spuriously return declined under gateway load.

allowed_tools / disallowed_tools are normalized (Bash → shell, Write/Edit → apply_patch, etc.) and passed as enabled_tools / disabled_tools in the thread config. Note: the Codex SDK does not currently enforce disabled_tools; do not rely on it as a security boundary (the agent logs a warning when it is set).

Skills Discovery

The agent sets up SKILL.md files (Agent Skills open standard) in .agents/skills/ directory:

  1. Scans config.plugins for local plugins with path field
  2. Checks plugin_tools_dir parameter passed to start()
  3. Expands environment variables in paths ($PLUGIN_PATH, ${PLUGIN_PATH})
  4. Symlinks skill directories (falls back to copying if symlink fails)
  5. Codex CLI auto-discovers skills in .agents/skills/

Async Integration

The Codex SDK is synchronous. The agent uses _run_async() helper to detect and await coroutines, preserving the async interface.

Differences from Claude Code Agent

Feature Claude Code Codex
SDK Type Subprocess (CLI via JSON generator) Sync client (app-server subprocess)
Command Tracking Full telemetry (tool name, params, duration) Streamed telemetry: shell → Bash, apply_patch → Write
Model Selection Direct via --model or config agent.model pinned into thread_start
Session Resume --resume {session_id} Via thread ID
Permissions permission_mode + allowed_tools permission_mode → sandbox/approval + allowed_tools/disallowed_tools → thread config
Tool Enforcement Not enforced by coder_eval wrapper enabled_tools honored; disabled_tools NOT enforced by the SDK

Known Limitations

  1. Tool-name collapse - Codex reports shell tools (Read/Grep/Bash) all as shell commands, surfaced as Bash telemetry; name-keyed criteria that distinguish these tools aren't meaningful across agents.
  2. skill_triggered criterion - Codex has no distinct Skill tool (it engages a skill by reading its files via shell), so the criterion detects Codex engagement from that file-read signal (a command referencing skills/<name>/) instead of a Skill tool call. The file-read signal is weaker than Claude's explicit invocation.
  3. disallowed_tools - passed to the SDK but not enforced; not a security boundary.
  4. Authentication - Requires CODEX_API_KEY in the environment (point it at whichever endpoint's key you use — OpenAI, gateway, or Azure); the agent calls login_api_key when a key is present. OPENAI_API_KEY/AZURE_OPENAI_API_KEY are NOT read.
  5. Model field - TurnRecord.model_used reflects the pinned agent.model; the Codex Turn payload itself doesn't carry the resolved model.
  6. Skills with Windows paths - Symlink creation may fail on Windows; agent falls back to copying (slower).

Future Enhancements

  • Implement session-based resume (thread ID tracking)
  • Strengthen the Codex skill_triggered signal — it currently infers engagement from a file read, weaker than Claude's Skill tool call
  • Capture the resolved model from the SDK (vs. the pinned config value)

Testing

Run the included test tasks:

# Basic functionality test
coder-eval run tasks/agents/codex_hello_world.yaml

# Tool restriction test (verifies disallowed_tools enforcement)
coder-eval run tasks/agents/codex_disallowed_tools_test.yaml

# Skills discovery test (requires PLUGIN_PATH environment variable)
export PLUGIN_PATH=~/path/to/skills
coder-eval run tasks/agents/codex_skills_test.yaml

Example unit test to verify agent setup:

import pytest
from coder_eval.models import AgentKind, AgentConfig
from coder_eval.agents.codex_agent import CodexAgent
from coder_eval.agent import AgentState

def test_codex_agent_initialization():
    """Verify CodexAgent can be instantiated with valid config."""
    config = AgentConfig(
        type=AgentKind.CODEX,
        permission_mode="acceptEdits",
        allowed_tools=["Bash", "Read", "Write"],
    )
    agent = CodexAgent(config)
    assert agent.get_state() == AgentState.WORKING
    assert agent.config.type == AgentKind.CODEX

def test_tool_name_mapping():
    """Verify Claude Code tool names map to Codex SDK names."""
    from coder_eval.agents.codex_agent import _CLAUDE_TO_CODEX_TOOL_MAP

    assert _CLAUDE_TO_CODEX_TOOL_MAP["Bash"] == "shell"
    assert _CLAUDE_TO_CODEX_TOOL_MAP["Write"] == "apply_patch"
    assert _CLAUDE_TO_CODEX_TOOL_MAP["Edit"] == "apply_patch"
    assert _CLAUDE_TO_CODEX_TOOL_MAP["Read"] == "shell"

def test_permission_mode_mapping():
    """Verify permission_mode maps to a sandbox; approval is uniformly deny_all."""
    from coder_eval.agents.codex_agent import (
        _CODEX_APPROVAL_MODE,
        _PERMISSION_MODE_TO_SANDBOX,
    )

    assert _PERMISSION_MODE_TO_SANDBOX["acceptEdits"] == "workspace-write"
    assert _PERMISSION_MODE_TO_SANDBOX["plan"] == "read-only"
    # Approval is the same for every permission mode — no per-mode mapping.
    assert _CODEX_APPROVAL_MODE == "deny_all"

References