Omnigent vendor harnesses on the BenchFlow gateway (codex unblock + claude-native)

[Yiminnn]

Context: most of this feature already shipped in #37 (omnigent-pi fixed on main; omnigent-claude verified reward 1.0 via the gateway). This PRD captures the full feature and specs the remaining forward-looking work: unblock omnigent-codex and wire omnigent-claude-native.

Problem Statement

A benchmark operator wants to run every Databricks Omnigent harness as a BenchFlow agent — on the benchmark's model, through the BenchFlow provider gateway, with usage captured — exactly like the ACP agents in this repo. Today only omnigent-pi and omnigent-claude actually run that way; omnigent-codex is wired but blocked, the native drivers don't surface a scoreable run, and the registry historically over-claimed (it listed 22 harnesses as "wireable" when the pinned omnigent only exposes 8 --harness values).

Solution

From the operator's perspective: pick any omnigent-<harness> agent and have it run a task end-to-end on the gateway, scoring a real reward with the raw llm_trajectory captured — or, if a harness can't run yet, have the registry tell them honestly why (blocked upstream / WIP / upstream-only) instead of failing opaquely. A vendor harness drives its own CLI but routes that CLI at the BenchFlow gateway (no vendor subscription, no foreign model).

User Stories

1. As a benchmark operator, I want omnigent-pi to run on main, so that the canonical omnigent harness scores end-to-end. (done — feat(omnigent): host only shipped harnesses + uniform gateway routing (#38, #39) #37)
2. As a benchmark operator, I want omnigent-claude to run the benchmark model through the gateway's Anthropic /v1/messages route, so that I can benchmark the Claude Code harness with captured usage. (done — feat(omnigent): host only shipped harnesses + uniform gateway routing (#38, #39) #37)
3. As a benchmark operator, I want omnigent-codex to run the benchmark model through the gateway, so that I can benchmark the Codex harness.
4. As a benchmark operator, I want omnigent-codex-native to run, so that I can benchmark omnigent's native Codex driver.
5. As a benchmark operator, I want omnigent-claude-native to surface a scoreable run, so that I can benchmark omnigent's native Claude driver.
6. As a benchmark operator, I want each registered agent's description to state its real status (WORKED / blocked / WIP / listed / upstream-only), so that I don't waste a run on an agent that can't launch.
7. As a benchmark operator, I want vendor harnesses to use the benchmark model (not the CLI's built-in default), so that cross-harness comparisons are fair.
8. As a benchmark operator, I want vendor harnesses to route through the proxy (not a vendor subscription), so that token usage and the raw llm_trajectory are captured.
9. As a benchmark operator, I want runs to behave identically across docker and daytona, so that the sandbox backend doesn't change results.
10. As a maintainer, I want the harness status tracked in one place (_HARNESS_STATUS), so that flipping a harness to WORKED is a one-line change.
11. As a maintainer, I want the registry to flag harnesses the pinned omnigent rejects, so that the upstream-derived list stays forward-looking without lying about what runs.
12. As a reviewer, I want the benchflow-experiment-review "unhealthy / missing tool usage metadata" result to be understood as omnigent's inherent coarse-trajectory limitation, so that a real reward isn't mistaken for a failure.

Implementation Decisions

• Vendor-CLI gateway routing lives in OmnigentAgent.connect. Each vendor harness installs its own CLI (per-harness install snippet in the registry) and connect writes that CLI's provider config / env from the resolved BENCHFLOW_PROVIDER_*:
  • claude (done): ANTHROPIC_BASE_URL (resolved base with trailing /v1 stripped — the Anthropic client appends /v1/messages), ANTHROPIC_AUTH_TOKEN, ANTHROPIC_MODEL; pass --model so the benchmark model is used.
  • codex (blocked): a ~/.codex/config.toml custom provider on the OpenAI responses wire with supports_websockets=false; omit --model (codex reads it from config; rejects a provider/model slug).
• The codex blocker is gateway-side, not the adapter. codex is responses-only (the OpenAI chat wire was removed); the gateway (litellm) serves /v1/chat/completions + /v1/messages but its /v1/responses route 404s the model (Model Group … NotFound). Decision: unblock by registering the benchmark model on the gateway's litellm /v1/responses route in benchflow-core — verified locally that a litellm 1.89 proxy with the model on the responses route lets omnigent run --harness codex return a result.
• claude-native uses omnigent's own driver; it launches with the same ANTHROPIC_* env but surfaces no scoreable run. Decision: treat as a separate investigation (the native driver's model/IO path differs from claude-sdk).
• Per-harness status is data (_HARNESS_STATUS + _REAL_OMNIGENT_HARNESSES), rendered into each agent's description. Keep the full upstream list; flag values the pinned omnigent rejects as upstream-only.
• Model handling: only codex omits --model (config-driven); pi and claude pass the benchmark model on omnigent run --model.

Testing Decisions

• A good test asserts external behavior: that an agent registers with the right harness value + status and that a harness runs end-to-end and scores — not internal config-string shapes.
• Unit seam (existing): omnigent/tests/test_register.py — registration, per-harness factory resolution, honest status text. Prior art: the existing test_register_includes_pi_and_claude_with_seam / test_per_harness_factories_*. Add a case per newly-wired harness asserting STATUS: WORKED + its CLI in install_cmd.
• Integration seam (existing, highest): bench eval create --include citation-check --agent omnigent-<h> --model deepseek/deepseek-v4-flash on docker + daytona, validated with benchflow-experiment-review (validate_run_artifacts.py). A harness is "wired" when it scores a real reward with llm_trajectory.jsonl present. Note the uniform n_tool_calls=0 "unhealthy" caveat is expected for omnigent.

Out of Scope

• The benchflow-core gateway change to serve deepseek-v4-flash on litellm's /v1/responses route (lives in benchflow, not this repo) — codex stays blocked here until that lands.
• The remaining upstream-only harnesses (cursor, opencode, hermes, goose, qwen, kimi, copilot, antigravity, + their *-native) — not exposed by omnigent 0.1.0; revisit when a newer omnigent ships them.
• Surfacing omnigent's per-tool-call stream (the n_tool_calls=0 coarse-trajectory limitation) — a larger omnigent-adapter rework.

Further Notes

• omnigent swallows the vendor CLI's stdout, so debug via an in-sandbox curl probe in connect (logger.warning), not the session trajectory. The interactive recipe: run a local litellm 1.89 proxy → deepseek and point the vendor CLI at it (omnigent run --harness <h> on the host) to see the real error.
• Verified in feat(omnigent): host only shipped harnesses + uniform gateway routing (#38, #39) #37: omnigent-pi reward 1.0 (docker), omnigent-claude reward 1.0 (docker + daytona); all runs capture the raw llm_trajectory.