feat(multi-agent): native concurrent floor (bench eval run --agents) + bf.* trajectory tree + hosted medical benchmark

[Yiminnn]

Summary

An opt-in, additive path for multi-agent on BenchFlow — the missing
inter-agent concurrency axis — taken from scaffold to real autonomous agents
competing live in one shared sandbox, plus the intra-agent trajectory tracking
that makes a shared proxy legible, plus a hosted multi-agent benchmark (medical).

Three additive layers (no scored single-agent path is touched):

1. Native concurrent multi-agent floor — bench eval run --agents (src/benchflow/arena/, benchmarks/casinobench/)

N real autonomous ACP agents play ONE shared environment concurrently, competing on
one leaderboard — reached through the standard run command, not a bespoke verb.

• Decoupled roster — --agents roster.yaml is the file form of repeated
  --agent/--model (a pure agents list; rejects run-level keys). Task
  (--tasks-dir/--game), service (--environment-manifest), sandbox, out, drive,
  prompt all follow the normal eval run flags. --agents is mutually exclusive
  with --agent; arena run is a deprecated hidden alias.
• Seat/player id defaults to <agent>-<model> (e.g. codex-acp-gpt-5.5-0) so the
  floor + viewer identify agent + model at a glance.
• In-sandbox env-0 service — the casino mock-service runs inside the shared
  rollout sandbox via ManifestEnvironment on localhost:9001 (no host
  subprocess — the earlier host-bridge design was removed). N seats share ONE
  service; each seat runs in /work/<seat> with its own acp_trajectory.jsonl, and
  API-key seats additionally get a per-seat LiteLLM proxy → raw llm_trajectory.jsonl
  (subscription/oauth seats go provider-direct, acp-only).
• benchmarks/casinobench — env-0 task package (environment.toml +
  casinobench-base image baking the engine [service] + the casino CLI). The
  deterministic engine/CLI/service live in the separate benchflow-ai/casinobench
  repo; this benchmark references them via the Docker build context.
• Animated town viewer — events.jsonl snapshot → replayable Stanford-Town board
  • click-to-open per-agent dossiers.

2. Multi-agent trajectory tracking — one proxy, structured tree (bf.*)

The fix for "a shared proxy yields one undifferentiated llm_trajectory": tag each
call with bf.* metadata and reconstruct the agent tree (OTel GenAI / LangSmith
parent_run_id shape).

• providers/litellm_logging.py — the callback records per-call
  bf.{agent_id,agent_name,span_kind,parent_agent_id,run_id,session_id,…}.
• trajectories/agents.py — build_agent_tree() rebuilds an unmixed AgentTree.
• Docs: docs/reference/multi-agent-trajectory.md + task-standard.md G7.

3. Hosted multi-agent benchmark — Multi-Agent Medical Assistant (benchmarks/medical-assistant/, examples/medical/)

souvikmajumder26/Multi-Agent-Medical-Assistant hosted on BenchFlow: a real
LangGraph supervisor→specialists graph (router → KB/web → confidence-gated handoff →
guardrail) run via ACP shim, each node tagged so the whole run reconstructs as one
unmixed tree — verified end-to-end on local, docker, and daytona.

Casino hardening (this branch)

Reward-integrity + robustness fixes found by inspecting real runs (casinobench-repo
commits, referenced via the build context):

• Secret world seed — the multiplayer World drew its seed from the agent-visible
  BENCHFLOW_SEED, letting a seat compute the deterministic roulette spin and bet the
  winning number (observed 26/27 straight-up, +45k). Now a secret os.urandom
  seed (CASINO_WORLD_SEED pins it for tests); BENCHFLOW_SEED no longer forwarded.
  Rerun confirms roulette back to ~fair (10–25% small-sample), max chips 2.4× not 46×.
• cwd-based seat id — a seat's CASINOBENCH_SEAT_ID defaults to its /work/<seat>
  folder, so it registers under its real name even when an agent runtime doesn't
  propagate the env (fixed the deepseek seat showing as agent).
• Heads-up turn timeout — a silent/absent actor at a 2-player table is auto-defaulted
  (was: the opponent pinned forever, no per-turn timeout).
• Viewer — reseat-reload fires at most once per roster (no infinite reload loop);
  dossier rows show the command.

Test plan

• arena/floor/trajectory suite green (162 passed on the branch).
• e2e (docker): 5 seats, one shared in-sandbox casino-service, separate per-seat
  trajectories, standings + reward vector, animated board — rerun after each fix.
• bf.* agent-tree unmixed on local + docker + daytona.
• daytona parity for the concurrent floor (docker-only verified) — follow-up.
• casino-server container isolation (cyberthon-style split; engine/seed unreadable
  by the agent) — follow-up.
• reconcile with [codex] Add CasinoBench-ready real multi-agent tracing #847's agent_graph.json trace layout — follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JDzpphBuqpjMH23dVJEoYX

[Yiminnn]

📦 Sample run results — multi-agent hosted by BenchFlow (proxy-tracked + trajectories)

Two real runs, each with every agent's raw LLM routed through a loopback BenchFlow LiteLLM proxy (ensure_litellm_runtime(environment="local")) — per-call usage/cost aggregated, raw exchanges persisted to <run>/trajectory/llm_trajectory.jsonl, and the raw provider key stripped from the agent (isolation invariant).

⬇️ Download the packed results folder: examples/sample_runs/proxy-runs.tar.gz (raw) — contains arena-floor-proxy/, medical-assistant/, each with run.log + trajectory/llm_trajectory.jsonl (+ per-seat trajectories), and a README.md.

Shared config

• model deepseek/deepseek-v4-pro (provider-prefixed → deepseek upstream) · proxy ensure_litellm_runtime(agent="deepagents", environment="local") · key DEEPSEEK_API_KEY (stripped from the agent by the proxy).

1) Inter-agent arena — 3 concurrent seats

set -a; . ./sb-run.env; set +a
uv run python examples/arena/run_through_proxy.py 3

→ 3 deepseek-v4 seats play one shared high-card round via run_arena; all reasoned to the dominant pick → tie → chips conserved 3000; proxy usage 2953 tokens, $0.0012 (usage_source=provider_response); per-seat decision trajectories + canonical llm_trajectory.jsonl.

2) Intra-agent LangGraph Medical-Assistant

uv pip install langgraph langchain-openai
set -a; . ./sb-run.env; set +a
MEDICAL_CONFIDENCE_THRESHOLD=1.01 \
  uv run python examples/medical/run_through_proxy.py "What are the main side effects of metformin?"

→ real langgraph.StateGraph, path supervisor → retrieve_kb → answer → web_search → answer → guardrail (confidence-gated handoff fired); proxy usage 1177 tokens, $0.00045; 4 node LLM calls.

Note: the proxy's canonical llm_trajectory.jsonl may show N-1 of N calls (LiteLLM async-callback flush timing on stop) — every call did route through the proxy, as the aggregated usage confirms; the per-seat/per-node decision trajectories are complete.

[Yiminnn]

Medical-Assistant bench — native vs BenchFlow-hosted

Same query (What are the main side effects of metformin?), same LangGraph (supervisor → specialists → guardrail), MEDICAL_CONFIDENCE_THRESHOLD=1.01 (forces the confidence-gated web handoff so answer fires twice).

	Native (direct deepseek)	Hosted (BenchFlow proxy)
Agent path	supervisor → retrieve_kb → answer → web_search → answer → guardrail	identical
route / confidence / safe	kb / 1.0 / True	kb / 0.9 / True
Answer	same clinical answer	same clinical answer
Usage / cost	none (no proxy → nothing tracked)	1293 tok · $0.00050, per agent
Raw llm_trajectory	none	one per agent (below)

Behaviour is identical; observability is the whole difference. Native gives you only the answer. Routing each specialist's LLM call through BenchFlow is what yields usage/cost + the raw-LLM trajectory, and the agent never sees the raw provider key.

Why the original hosted run was a single file — and the fix

LangGraph runs the nodes sequentially, and they shared one proxy → one callback log → one combined llm_trajectory.jsonl in call order. The callback records only model + messages (no per-call agent tag), so a shared log cannot be split after the fact. Giving each specialist its own proxy → its own log:

out/medical-hosted/supervisor/trajectory/llm_trajectory.jsonl  (1 call(s))
out/medical-hosted/answer/trajectory/llm_trajectory.jsonl  (2 call(s))
out/medical-hosted/guardrail/trajectory/llm_trajectory.jsonl  (1 call(s))

Entire running folder (out/medical-hosted/) — full records

supervisor/trajectory/llm_trajectory.jsonl — 1 call(s)

{
  "request": {
    "timestamp": "2026-06-29T00:15:28.120343",
    "method": "POST",
    "path": "/v1/chat/completions",
    "headers": {},
    "body": {
      "model": "deepseek-v4-pro",
      "messages": [
        {
          "content": "You route a medical question to a specialist. Question: 'What are the main side effects of metformin?'. Reply with ONE word: 'kb' if a drug/condition knowledge base likely covers it, else 'web'.",
          "role": "user"
        }
      ],
      "input": [
        {
          "content": "You route a medical question to a specialist. Question: 'What are the main side effects of metformin?'. Reply with ONE word: 'kb' if a drug/condition knowledge base likely covers it, else 'web'.",
          "role": "user"
        }
      ],
      "stream": false
    }
  },
  "response": {
    "timestamp": "2026-06-29T00:15:31.109319",
    "status_code": 200,
    "headers": {},
    "body": {
      "id": "252168fc-1e5b-441c-ba0a-b78ede67386b",
      "created": 1782692129,
      "model": "benchflow-deepseek-deepseek-v4-pro",
      "object": "chat.completion",
      "system_fingerprint": "fp_9954b31ca7_prod0820_fp8_kvcache_20260402",
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
            "content": "kb",
            "role": "assistant",
            "tool_calls": null,
            "function_call": null,
            "reasoning_content": "We are asked: \"You route a medical question to a specialist. Question: 'What are the main side effects of metformin?'. Reply with ONE word: 'kb' if a drug/condition knowledge base likely covers it, else 'web'.\"\n\nThe question is about side effects of a common drug, metformin. Knowledge bases (kb) typically contain structured information on drug side effects. This is a straightforward factual question that a drug database like those used in medical settings would cover. So the answer is 'kb'.",
            "provider_specific_fields": {
              "refusal": null,
              "reasoning_content": "We are asked: \"You route a medical question to a specialist. Question: 'What are the main side effects of metformin?'. Reply with ONE word: 'kb' if a drug/condition knowledge base likely covers it, else 'web'.\"\n\nThe question is about side effects of a common drug, metformin. Knowledge bases (kb) typically contain structured information on drug side effects. This is a straightforward factual question that a drug database like those used in medical settings would cover. So the answer is 'kb'."
            }
          },
          "provider_specific_fields": {}
        }
      ],
      "usage": {
        "completion_tokens": 108,
        "prompt_tokens": 50,
        "total_tokens": 158,
        "completion_tokens_details": {
          "accepted_prediction_tokens": null,
          "audio_tokens": null,
          "reasoning_tokens": 106,
          "rejected_prediction_tokens": null
        },
        "prompt_tokens_details": {
          "audio_tokens": null,
          "cached_tokens": 0
        },
        "prompt_cache_hit_tokens": 0,
        "prompt_cache_miss_tokens": 50
      },
      "service_tier": null
    }
  },
  "duration_ms": 2988.976001739502
}

answer/trajectory/llm_trajectory.jsonl — 2 call(s)

{
  "request": {
    "timestamp": "2026-06-29T00:15:32.348103",
    "method": "POST",
    "path": "/v1/chat/completions",
    "headers": {},
    "body": {
      "model": "deepseek-v4-pro",
      "messages": [
        {
          "content": "Question: What are the main side effects of metformin?\nEvidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\nAnswer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.",
          "role": "user"
        }
      ],
      "input": [
        {
          "content": "Question: What are the main side effects of metformin?\nEvidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\nAnswer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.",
          "role": "user"
        }
      ],
      "stream": false
    }
  },
  "response": {
    "timestamp": "2026-06-29T00:15:38.421150",
    "status_code": 200,
    "headers": {},
    "body": {
      "id": "20acd434-6593-4336-99bb-d668102357d0",
      "created": 1782692134,
      "model": "benchflow-deepseek-deepseek-v4-pro",
      "object": "chat.completion",
      "system_fingerprint": "fp_9954b31ca7_prod0820_fp8_kvcache_20260402",
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
            "content": "Metformin commonly causes gastrointestinal upset (nausea, diarrhea), which can be mitigated by slow dose titration and taking with food. Rarely, it may cause lactic acidosis, particularly in patients with significant renal impairment (contraindicated if eGFR <30 mL/min/1.73 m²).  \nCONFIDENCE: 1.0",
            "role": "assistant",
            "tool_calls": null,
            "function_call": null,
            "reasoning_content": "We are asked: \"Question: What are the main side effects of metformin? Evidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. Answer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.\"\n\nWe need to provide a concise answer for a clinician, summarizing the main side effects. The evidence given: \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" So answer: GI upset (nausea, diarrhea) are common; lactic acidosis is a rare but serious side effect. We can mention that it's generally well-tolerated, but GI effects often improve with dose titration. Also, lactic acidosis risk is increased in renal impairment, so contraindicated if eGFR <30. But the evidence only mentions those points. The answer should be concise.\n\nThen we output confidence score. The evidence directly supports the answer: it states common side effects and a rare one. Confidence high, like 1.0. But we need to interpret: The evidence says \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" This is accurate. So confidence: 1.0.",
            "provider_specific_fields": {
              "refusal": null,
              "reasoning_content": "We are asked: \"Question: What are the main side effects of metformin? Evidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. Answer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.\"\n\nWe need to provide a concise answer for a clinician, summarizing the main side effects. The evidence given: \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" So answer: GI upset (nausea, diarrhea) are common; lactic acidosis is a rare but serious side effect. We can mention that it's generally well-tolerated, but GI effects often improve with dose titration. Also, lactic acidosis risk is increased in renal impairment, so contraindicated if eGFR <30. But the evidence only mentions those points. The answer should be concise.\n\nThen we output confidence score. The evidence directly supports the answer: it states common side effects and a rare one. Confidence high, like 1.0. But we need to interpret: The evidence says \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" This is accurate. So confidence: 1.0."
            }
          },
          "provider_specific_fields": {}
        }
      ],
      "usage": {
        "completion_tokens": 350,
        "prompt_tokens": 86,
        "total_tokens": 436,
        "completion_tokens_details": {
          "accepted_prediction_tokens": null,
          "audio_tokens": null,
          "reasoning_tokens": 280,
          "rejected_prediction_tokens": null
        },
        "prompt_tokens_details": {
          "audio_tokens": null,
          "cached_tokens": 0
        },
        "prompt_cache_hit_tokens": 0,
        "prompt_cache_miss_tokens": 86
      },
      "service_tier": null
    }
  },
  "duration_ms": 6073.046922683716
}
{
  "request": {
    "timestamp": "2026-06-29T00:15:38.471754",
    "method": "POST",
    "path": "/v1/chat/completions",
    "headers": {},
    "body": {
      "model": "deepseek-v4-pro",
      "messages": [
        {
          "content": "Question: What are the main side effects of metformin?\nEvidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. [web] current guidance relevant to: What are the main side effects of metformin?\nAnswer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.",
          "role": "user"
        }
      ],
      "input": [
        {
          "content": "Question: What are the main side effects of metformin?\nEvidence: Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. [web] current guidance relevant to: What are the main side effects of metformin?\nAnswer concisely for a clinician. Then on a NEW line output 'CONFIDENCE: <0.0-1.0>' for how well the evidence supports your answer.",
          "role": "user"
        }
      ],
      "stream": false
    }
  },
  "response": {
    "timestamp": "2026-06-29T00:15:42.344671",
    "status_code": 200,
    "headers": {},
    "body": {
      "id": "df717fc6-808b-42e0-90d3-0f0893bfb76d",
      "created": 1782692138,
      "model": "benchflow-deepseek-deepseek-v4-pro",
      "object": "chat.completion",
      "system_fingerprint": "fp_9954b31ca7_prod0820_fp8_kvcache_20260402",
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
            "content": "The main side effects of metformin are gastrointestinal disturbances (notably nausea, diarrhea, and abdominal discomfort), which are common and often dose-dependent. A rare but serious adverse effect is lactic acidosis, particularly in patients with renal impairment or other contraindications. Long-term use can also lead to vitamin B12 deficiency.\n\nCONFIDENCE: 0.9",
            "role": "assistant",
            "tool_calls": null,
            "function_call": null,
            "reasoning_content": "We are asked: \"Question: What are the main side effects of metformin?\" with evidence: \"Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. [web] current guidance relevant to: What are the main side effects of metformin?\" Then answer concisely for a clinician, then on a new line output 'CONFIDENCE: <0.0-1.0>'.\n\nI need to answer based on the evidence. The evidence mentions GI upset (nausea, diarrhea) and rarely lactic acidosis. That's it. So I'll answer: common GI side effects (nausea, diarrhea, abdominal discomfort) and rare lactic acidosis, especially in renal impairment. I'll add that B12 deficiency can also occur with long-term use, but the evidence doesn't mention it, but it's well-known. However, the instruction is to answer from the evidence provided. The evidence says \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" So I'll stick to that. I'll phrase it for a clinician. Then confidence: The evidence is from a web source, probably reliable, but I'll rate confidence high maybe 0.9. I'll say CONFIDENCE: 0.9.",
            "provider_specific_fields": {
              "refusal": null,
              "reasoning_content": "We are asked: \"Question: What are the main side effects of metformin?\" with evidence: \"Metformin is first-line for type 2 diabetes; common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis. [web] current guidance relevant to: What are the main side effects of metformin?\" Then answer concisely for a clinician, then on a new line output 'CONFIDENCE: <0.0-1.0>'.\n\nI need to answer based on the evidence. The evidence mentions GI upset (nausea, diarrhea) and rarely lactic acidosis. That's it. So I'll answer: common GI side effects (nausea, diarrhea, abdominal discomfort) and rare lactic acidosis, especially in renal impairment. I'll add that B12 deficiency can also occur with long-term use, but the evidence doesn't mention it, but it's well-known. However, the instruction is to answer from the evidence provided. The evidence says \"common side effects are GI upset (nausea, diarrhea) and, rarely, lactic acidosis.\" So I'll stick to that. I'll phrase it for a clinician. Then confidence: The evidence is from a web source, probably reliable, but I'll rate confidence high maybe 0.9. I'll say CONFIDENCE: 0.9."
            }
          },
          "provider_specific_fields": {}
        }
      ],
      "usage": {
        "completion_tokens": 346,
        "prompt_tokens": 104,
        "total_tokens": 450,
        "completion_tokens_details": {
          "accepted_prediction_tokens": null,
          "audio_tokens": null,
          "reasoning_tokens": 274,
          "rejected_prediction_tokens": null
        },
        "prompt_tokens_details": {
          "audio_tokens": null,
          "cached_tokens": 0
        },
        "prompt_cache_hit_tokens": 0,
        "prompt_cache_miss_tokens": 104
      },
      "service_tier": null
    }
  },
  "duration_ms": 3872.9169368743896
}

guardrail/trajectory/llm_trajectory.jsonl — 1 call(s)

{
  "request": {
    "timestamp": "2026-06-29T00:15:44.866217",
    "method": "POST",
    "path": "/v1/chat/completions",
    "headers": {},
    "body": {
      "model": "deepseek-v4-pro",
      "messages": [
        {
          "content": "Output guardrail. The assistant answered: 'The main side effects of metformin are gastrointestinal disturbances (notably nausea, diarrhea, and abdominal discomfort), which are common and often dose-dependent. A rare but serious adverse effect is lactic acidosis, particularly in patients with renal impairment or other contraindications. Long-term use can also lead to vitamin B12 deficiency.'. Is it safe and appropriately cautious as medical information (not a diagnosis)? Reply yes/no.",
          "role": "user"
        }
      ],
      "input": [
        {
          "content": "Output guardrail. The assistant answered: 'The main side effects of metformin are gastrointestinal disturbances (notably nausea, diarrhea, and abdominal discomfort), which are common and often dose-dependent. A rare but serious adverse effect is lactic acidosis, particularly in patients with renal impairment or other contraindications. Long-term use can also lead to vitamin B12 deficiency.'. Is it safe and appropriately cautious as medical information (not a diagnosis)? Reply yes/no.",
          "role": "user"
        }
      ],
      "stream": false
    }
  },
  "response": {
    "timestamp": "2026-06-29T00:15:50.276964",
    "status_code": 200,
    "headers": {},
    "body": {
      "id": "576d4dcb-0170-40b5-990d-d85895178409",
      "created": 1782692147,
      "model": "benchflow-deepseek-deepseek-v4-pro",
      "object": "chat.completion",
      "system_fingerprint": "fp_9954b31ca7_prod0820_fp8_kvcache_20260402",
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
            "content": "yes",
            "role": "assistant",
            "tool_calls": null,
            "function_call": null,
            "reasoning_content": "We are asked to evaluate if the assistant's answer is safe and appropriately cautious as medical information (not a diagnosis). The answer states: \"The main side effects of metformin are gastrointestinal disturbances (notably nausea, diarrhea, and abdominal discomfort), which are common and often dose-dependent. A rare but serious adverse effect is lactic acidosis, particularly in patients with renal impairment or other contraindications. Long-term use can also lead to vitamin B12 deficiency.\"\n\nThis seems factually accurate, mentions common side effects, a rare but serious side effect, and a long-term risk. It doesn't provide a diagnosis, just lists side effects. It's cautious by mentioning the seriousness of lactic acidosis and the context of renal impairment. So yes, it's safe and appropriately cautious.",
            "provider_specific_fields": {
              "refusal": null,
              "reasoning_content": "We are asked to evaluate if the assistant's answer is safe and appropriately cautious as medical information (not a diagnosis). The answer states: \"The main side effects of metformin are gastrointestinal disturbances (notably nausea, diarrhea, and abdominal discomfort), which are common and often dose-dependent. A rare but serious adverse effect is lactic acidosis, particularly in patients with renal impairment or other contraindications. Long-term use can also lead to vitamin B12 deficiency.\"\n\nThis seems factually accurate, mentions common side effects, a rare but serious side effect, and a long-term risk. It doesn't provide a diagnosis, just lists side effects. It's cautious by mentioning the seriousness of lactic acidosis and the context of renal impairment. So yes, it's safe and appropriately cautious."
            }
          },
          "provider_specific_fields": {}
        }
      ],
      "usage": {
        "completion_tokens": 155,
        "prompt_tokens": 94,
        "total_tokens": 249,
        "completion_tokens_details": {
          "accepted_prediction_tokens": null,
          "audio_tokens": null,
          "reasoning_tokens": 153,
          "rejected_prediction_tokens": null
        },
        "prompt_tokens_details": {
          "audio_tokens": null,
          "cached_tokens": 0
        },
        "prompt_cache_hit_tokens": 0,
        "prompt_cache_miss_tokens": 94
      },
      "service_tier": null
    }
  },
  "duration_ms": 5410.747051239014
}

[Yiminnn]

Hosted as a BenchFlow benchmark — run in Docker (deepseek-v4-pro via the proxy)

Per the benchmarks/ convention, the Multi-Agent-Medical-Assistant is now a first-class benchmark: a registered medical-assistant ACP agent (the LangGraph supervisor→specialists graph) + 3 Harbor drug-safety tasks. Each rollout runs in its own Docker sandbox (environment: docker), agent sandboxed as non-root agent, every node's LLM routed through the LiteLLM proxy.

Result: 3/3 (100%), errors=0, 3.2 min.

Task	Reward	Agent path (per-node, from acp_trajectory)	LLM calls	Tokens	Cost
aspirin-secondary-prevention	1.0	supervisor → retrieve_kb → answer → guardrail	3	919	$0.00035
ibuprofen-renal-caution	1.0	supervisor → retrieve_kb → answer → guardrail	3	1395	$0.00055
metformin-side-effects	1.0	supervisor → retrieve_kb → answer → guardrail	3	726	$0.00027
total	3/3			3040	$0.00117

Every rollout captured both trajectories under <rollout>/trajectory/:

• llm_trajectory.jsonl — the proxy's raw-LLM log (3 calls: supervisor, answer, guardrail; retrieve_kb is non-LLM).
• acp_trajectory.jsonl — the multi-agent structure as per-node ACP tool_call steps, so which specialist ran (and in what order) is visible even though the raw calls share one proxy log.

Usage is usage_source=provider_response (proxy-metered), and the agent never saw the raw provider key.

Note on the verifier

First run scored ibuprofen 0.5: the answer was clinically correct (named both real cautions — renal impairment + anticoagulant/bleeding) but the verifier required the drug-class token "NSAID" rather than the cautions the question asks for. The verifier was corrected to test the two actual cautions (matches the task KB); the re-run above is the honest harness result.

Run it

set -a; . ~/sb-run.env; set +a
python benchmarks/medical-assistant/run_medical_assistant.py
# or: bench eval run --config benchmarks/medical-assistant/medical-assistant-deepseek.yaml

[Yiminnn]

Casino floor — 10 concurrent real agents, all 18 games

Live realization of arena-concurrent: 10 autonomous ACP agents played one shared
casinobench World concurrently, each in its own DockerSandbox, choosing games freely
via the casino CLI. Models are limited to what the subscriptions expose
(codex→gpt-5.5; claude→sonnet-4-6/haiku-4-5 — gpt-5*/opus-4-8 are rejected by the plans).

Rank	Seat	Model	Chips	Δ	Moves	Status
1	codex-gpt-5.5-0	gpt-5.5	1700	+700	51	finished
2	codex-gpt-5.5-3	gpt-5.5	1275	+275	30	timed out (still played)
3	codex-gpt-5.5-2	gpt-5.5	1032	+32	28	finished
4	claude-sonnet-4-6-2	claude-sonnet-4-6	996	-4	67	timed out (still played)
5	claude-haiku-4-5-2	claude-haiku-4-5	953	-47	55	timed out (still played)
6	claude-haiku-4-5-1	claude-haiku-4-5	900	-100	34	finished
7	claude-haiku-4-5-0	claude-haiku-4-5	850	-150	60	finished
8	claude-sonnet-4-6-1	claude-sonnet-4-6	751	-249	52	timed out (still played)
9	claude-sonnet-4-6-0	claude-sonnet-4-6	550	-450	40	timed out (still played)
10	codex-gpt-5.5-1	gpt-5.5	0	-1000	72	finished

Total chips on the floor: 9007 (started 10000; the rest went to the house games).

Per-seat trajectories captured under out/casino-floor/all-games/<seat>/trajectory/ —
subscription seats produce acp_trajectory.jsonl (their casino moves + thinking).
Each agent's run is browsable in the live town viewer's click-to-open dossier.

[Yiminnn]

New on this branch: native concurrent multi-agent runner (benchflow arena run --agents agents.yaml)

Commit 98ae7a6 adds the framework-level feature requested separately from the medical scope above — N agents on ONE shared task + its ONE service, concurrently, in ONE shared sandbox, each agent in its own /work/<seat> folder, with per-agent ACP + raw-LLM trajectories in separate files. It builds on this branch's arena concurrency scaffold and reuses connect_acp, the per-seat LiteLLM proxy, and the trajectory writer — no new concurrency/trajectory machinery.

Run it

set -a; . ~/sb-run.env; set +a
uv run benchflow arena run --agents examples/casino/agents.yaml

What's in it

File	Role
arena/agents_manifest.py	pydantic agents.yaml parser, count fan-out, agent: (prebuilt) XOR manifest: (BYOA), strict ACP-contract BYOA loader → register_agent, unique seat ids
arena/agent_driver.py	protocol-branched connect/prompt/close — acp/acpx→connect_acp+execute_prompts (covers raw ACP and ai-sdk); session-factory→Agent.connect/Session.prompt (omnigent). import_callable, never eval
arena/instructions.py + AgentConfig.instruction_filename	write each seat's instructions: into /work/<seat>/{CLAUDE.md|GEMINI.md|AGENTS.md} before launch
arena/concurrent_floor.py	shared-sandbox orchestrator (asyncio.gather), subscription-vs-proxy by actual auth (uses_native_subscription_auth), auto-loop + service-rounds (mock-service-driven rounds), per-seat try/except, separate acp_/llm_ trajectory files + roster.json/floor.json
arena/bootstrap.py + cli/arena.py	benchflow arena run --agents seeds ONE shared sandbox + host service, runs, tears down; opt-in standings→SharedEnvReward vector
examples/casino/agents.yaml + prompts/, examples/arena/README.md	worked example + docs

Output layout

out/native-floor/casino/
├── roster.json                       # seat → agent / model / protocol / byoa
├── floor.json                        # per-seat status + raw flag + (opt) standings + reward
└── <seat>/trajectory/
    ├── acp_trajectory.jsonl          # every seat
    └── llm_trajectory.jsonl          # PROXY seats only (oauth seats are acp-only by design)

Three agent paths, one config

raw ACP / ai-sdk / omnigent (prebuilt) or a BYOA manifest.toml all collapse to one AgentConfig. acp is the verified path (covers ai-sdk too); session-factory is structural/unverified — no session-factory agent is registered in this repo yet, so it's tested with a fake.

Multi-round (answering "can the mock server trigger multiple rounds?")

Yes. execute_prompts is re-entrant. drive: auto-loop (default) runs multi-round inside one prompt (the agent owns its observe→act loop); drive: service-rounds has the service drive the rounds — the runner polls per seat and re-prompts (nudges) only on YOUR_TURN, until DONE/deadline.

Tests

26 new tests (tests/test_{agents_manifest,agent_driver,agent_instructions,concurrent_floor,arena_cli}.py), all green; broad arena/registry regression green; ruff clean. (One pre-existing repo test, test_cli_verify_nonexistent_roundtrip, fails identically on clean HEAD — unrelated, hits an LLM judge.)

A code review pass was run; all findings addressed — notably the subscription gate now keys on actual auth (an API-key claude/codex seat correctly takes the proxy+raw path, not forced oauth-only), raw is reported only when a proxy actually ran, session-factory timeouts are non-fatal, and duplicate seat names are rejected.

Deliberate scope notes

• Kept the working examples/casino/run_floor.py (per-agent-sandbox live-viewer demo) untouched; added the shared-sandbox native path alongside it (same roster.json/<seat>/trajectory/ layout, so the viewer still works).
• Deferred (noted, not built): daytona shared-sandbox parity (the CLI fails closed on --environment daytona), #847's exact trajectory/agents/<seat>/ nesting + agent_graph.json, CI smoke for the baked image.

Happy to split this into its own PR or retitle 846 if you'd prefer to keep the medical scope clean — just say which.

[Yiminnn]

✅ Validated end-to-end on docker — real 5-agent concurrent run

Ran benchflow arena run --agents examples/casino/agents.yaml for real: 5 agents, ONE shared docker sandbox, ONE casino World, concurrent, drive: auto-loop. 5/5 seats played. Numbers below were re-derived independently from the on-disk artifacts.

seat	agent / model	status	acp tool-calls	acp traj	raw llm traj
codex-0	codex-acp / gpt-5.5	ok	55	99 lines	— (acp-only)
codex-1	codex-acp / gpt-5.5	ok	90	133 lines	— (acp-only)
claude-0	claude-agent-acp / sonnet-4-6	timeout (played 41)	41	83 lines	— (acp-only)
claude-1	claude-agent-acp / sonnet-4-6	timeout (played 69)	69	188 lines	— (acp-only)
deepseek	deepagents / deepseek-v4-pro	ok	67	70 lines	45 records · 5.5 MB

Standings (chips, start ≈1000) → reward vector: codex-0 2246 (+1246) · claude-0 700 (-300) · codex-1 604 (-396) · claude-1 350 (-650).

What this proves

• One shared sandbox, per-seat /work/<seat> folders — all 5 agents connected concurrently into the single container.
• Separate per-agent trajectories — 5 distinct acp_trajectory.jsonl; the proxy seat is the only one with a raw llm_trajectory.jsonl (model deepseek-v4-pro) — zero cross-seat leakage.
• Mixed-raw by actual auth — codex/claude oauth seats → acp-only; the API-key proxy seat → raw+acp (the review-fix gate, in the wild).
• timeout ≠ error — 2 claude seats hit the 20-min wall clock, classified timeout (played N), not failures; 3 finished ok.
• Per-agent instruction files written into each cwd; roster.json + floor.json (+ reward vector) emitted.

Two real bugs the run surfaced — fixed (74a3429) and validated by a clean re-run

1. Shared-sandbox env-file race — DockerProcess used a fixed /tmp/.benchflow_env; concurrent agents' source && rm deleted each other's file (No such file or directory, ACP rc=1). → unique /tmp/.benchflow_env_<uuid> per process (+ trap-based cleanup), regression-tested.
2. Per-seat id env — added services.seat_env (→ CASINOBENCH_SEAT_ID) so the service identifies each player.

Follow-up hardening (6666f51)

• Host-service teardown now reaps the whole process group (uvicorn workers no longer outlive the run).
• Example's proxy seat swapped deepagents → openhands (still a proxy seat → raw+acp; model reaches it via LLM_MODEL on the proxy path).

Known gap (not a framework bug)

The proxy seat is absent from the casino standings/reward maps even though it played and its trajectories were captured — the casino service's chip-accounting didn't include it (gameplay/service-side, not the runner). Trajectory separation — the feature under test — is unaffected.

Test status: 28 new tests green; broad arena/registry/sandbox regression green; ruff clean.

[bingran-you]

Daily scan status (2026-07-03): current head f5ef1587 is still draft, conflict/dirty against main, and has no current CI checks on the latest head. Labels remain appropriate (status:blocked, review:changes-requested, area:eval). Next action is still author-side: resolve conflicts, keep scope tight after the related split work, get fresh CI, then request re-review.

[bingran-you]

Users Simulation review (2026-07-03): blocked.

The draft remains non-mergeable. Targeted checks passed (bench eval run --help, plus tests/test_agent_driver.py tests/test_agents_manifest.py tests/test_arena.py tests/test_arena_cli.py tests/test_concurrent_floor.py tests/test_litellm_logging.py tests/trajectories/test_agent_tree.py -> 38 passed), but user-simulation probes found release-blocking regressions:

• --drive service-rounds crashes on the default path because run_native_floor() never wires an HTTP client and HttpSeatClient dereferences None.
• bench eval run --agents drops run-level controls such as --reasoning-effort and --agent-idle-timeout, so multi-agent runs silently ignore user settings.
• The hidden/deprecated arena run alias passes multiplayer= into run_floor_from_cli, which raises TypeError.

No live trajectory was generated, so the new bf.* trajectory tree and Prime-RL readiness still need real artifact validation after these CLI/runtime blockers are fixed.

[bingran-you]

Automation status refresh (2026-07-04): current head 676c1c9 is still draft, dirty/conflicting against main, and has no current CI checks. Labels remain correct: status:blocked, review:changes-requested, area:eval. The standing blockers still require author-side work before any merge review: resolve conflicts, split/trim the broad arena/medical scope, fix the known multi-agent runtime regressions (service-rounds HTTP client path, dropped run-level controls, deprecated arena alias), then attach fresh CI plus real trajectory validation evidence.

[bingran-you]

Users Simulation automation review (2026-07-04): still blocked on current draft head 676c1c9c.

Current-head user simulation found real CLI contract blockers:

• benchflow arena run --agents examples/casino/roster-smoke-oc.yaml --environment-manifest benchmarks/casinobench/environment.toml --out /tmp/benchflow-pr846-arena-alias-smoke still crashes with TypeError: run_floor_from_cli() got an unexpected keyword argument 'multiplayer'.
• The bench eval run --agents ... --drive service-rounds --reasoning-effort max --agent-idle-timeout 0 --usage-tracking required branch still drops --reasoning-effort, --agent-idle-timeout, and --usage-tracking before the floor runner.
• Roster.from_yaml('examples/casino/agents.yaml') still rejects the documented full-manifest shape (sandbox/services/out/drive/deadline_s/idle_timeout_s); examples/casino/roster.yaml is the valid shape, so docs/examples are stale.
• The diff is still broad for review: 112 files / 7,416 insertions, on top of the behavior bugs.

Not currently counted as a blocker: an in-memory service-rounds client probe passed (GET /observe, POST /act, 2 prompts). Shared sweep canary on current main produced healthy ACP+LLM/results artifacts, but this PR is not ready. Keeping status:blocked / review:changes-requested.

[bingran-you]

Users Simulation automation review (2026-07-05): blocked on head 676c1c9c7e9cca1be992f84828a6300d4caeb7ae.

Fresh isolated validation reproduced the merge blockers:

• bench arena run ... still crashes with TypeError: run_floor_from_cli() got an unexpected keyword argument 'multiplayer'.
• bench eval run --agents still drops run-level knobs: --reasoning-effort, --usage-tracking, and --agent-idle-timeout.
• tests/test_cli_docs_drift.py::test_eval_run_flags_match_cli_md_bidirectional still fails.
• examples/casino/agents.yaml is rejected by Roster.from_yaml() because it contains run-level keys.
• git merge-tree --write-tree --messages origin/main HEAD still conflicts in src/benchflow/agents/registry.py.
• Thermo-nuclear check: docs/task-standard.md crosses the 1k-line threshold (970 -> 1008 lines).

Labels: keep status:blocked + review:changes-requested.

[bingran-you]

Users Simulation automation review (2026-07-06): still blocked on draft/dirty PR head 676c1c9c.

Blockers reproduced:

• bench arena run --agents ... fails with TypeError: run_floor_from_cli() got an unexpected keyword argument 'multiplayer'.
• ruff check fails on an unused noqa; ty check catches the stale multiplayer kwarg and a type-unsafe sandbox bootstrap forwarding path.
• The new arena/floor surfaces did not prove the required training artifact contract; the checked path only writes roster.json / floor.json, and no completed rollout root with results.jsonl was available.

Keep status:blocked / review:changes-requested until the stale alias kwarg, static-check failures, and artifact-output proof are fixed.

[bingran-you]

Users Simulation automation review (2026-07-07): still blocked on current draft head 676c1c9c7e9cca1be992f84828a6300d4caeb7ae.

Fresh isolated validation added one more current-head blocker on top of the existing draft/dirty/no-current-CI state:

• Focused unit coverage passed for concurrent floor, agent manifest, agent-tree trajectory, base import, and smoke-wiring surfaces.
• examples/casino/roster-10.yaml parsed as 10 seats.
• A bench eval run --agents casino smoke under amd64 emulation wrote floor.json, events.jsonl, and per-seat ACP/LLM trajectory files, but the run was not healthy: one seat failed with ACPError: Authentication required and another timed out without moves. The normal benchflow-experiment-review rollout validator reported checked: 0 because this floor root is not a standard rollout tree with result.json / rollout-level results.jsonl.
• New blocker: examples/casino/PROMPT.md is not wired into the live runner. examples/casino/run_floor.py still uses a hardcoded prompt, so the newly added leave / --wait guidance never reaches the actual bench eval run --agents path. Please make the runner load one canonical prompt source or delete the dead markdown file.

Keep status:blocked / review:changes-requested. This PR still needs conflict cleanup, fresh CI, a healthy standard artifact story for the new floor trajectory shape, and the prompt-wiring fix before merge review.

[bingran-you]

Users Simulation automation review (2026-07-08): blocked on head 676c1c9c7e9cca1be992f84828a6300d4caeb7ae.

Fresh blockers:

• Deprecated arena alias path is broken. uv run python -m benchflow.cli.main arena run --agents ... --environment-manifest benchmarks/casinobench/environment.toml --out ... fails with TypeError: run_floor_from_cli() got an unexpected keyword argument 'multiplayer'.
• Casino prompt source is still split: examples/casino/PROMPT.md exists, but examples/casino/run_floor.py hardcodes its own prompt instead of reading the file. That means user-visible prompt edits can drift from the actual floor run.
• The floor runner writes roster.json, floor.json, and per-seat trajectories, but not the standard rollout result.json / results.jsonl surface; I would not call that path publishable under the current artifact-health bar.

Passing evidence:

• uv sync --extra dev --locked passed.
• Focused tests passed: tests/test_arena_cli.py, tests/test_concurrent_floor.py, tests/trajectories/test_agent_tree.py, tests/test_litellm_logging.py (20 passed).
• Unit-level bf-tag/tree reconstruction is healthy: bf.* is captured into request metadata and the agent forest is reconstructed without mixing agents.

Thermo-nuclear review: the real issue is integration-boundary mess, not the bf-tree code itself. The alias crash plus prompt-source duplication should be fixed before this draft can be considered mergeable.

Labels: kept status:blocked / review:changes-requested.

[bingran-you]

Users Simulation automation review (2026-07-10): still blocked on head 676c1c9c7e9cca1be992f84828a6300d4caeb7ae.

Live gate remains unchanged: this PR is still draft and conflicting, so no merge-readiness decision can clear it.

Cheap checks in an isolated worktree passed after syncing dev deps:

• benchflow --help
• benchflow arena --help
• Python import smoke
• arena/agent/trajectory focused tests (tests/test_arena_cli.py, tests/test_agent_driver.py, tests/test_roster.py, tests/test_agents_manifest.py, tests/test_concurrent_floor.py, tests/test_litellm_logging.py, tests/trajectories/test_agent_tree.py, tests/test_capture_trajectory.py, tests/environment/test_manifest_env.py, tests/test_docker_process_env_isolation.py) — 97 passed
• trajectory/protocol focused tests — 25 passed

Thermo-nuclear blockers remain:

• src/benchflow/arena/concurrent_floor.py mixes generic service-round orchestration with casino-specific observe/nudge behavior (/casino/observe, casino notifications, keep-playing prompts). That benchmark-specific behavior should sit behind an adapter, not in the shared floor runner.
• src/benchflow/arena/bootstrap.py silently selects manifest.services[0] as the shared service. Require exactly one service or select by name so multi-service manifests cannot point the floor at the wrong endpoint.
• The shipped medical-assistant registry path uses src/benchflow/agents/medical_acp_shim.py, while the bf.* tree-tagged logic lives in examples/medical/medical_assistant.py; the benchmark path and example path diverge on trajectory semantics. Please share one medical graph implementation.

Keeping status:blocked / review:changes-requested.

[bingran-you]

Users Simulation automation review (2026-07-11): still blocked on head 676c1c9c7e9cca1be992f84828a6300d4caeb7ae.

Live PR state is still not mergeable: draft, mergeStateStatus=DIRTY, labels status:blocked / review:changes-requested, and no status-check rollup.

Cheap deterministic checks:

• Focused arena tests passed: 41 tests in the subagent pass; 23 tests in the main-thread slice.
• ruff check tests/test_arena_bootstrap.py fails at tests/test_arena_bootstrap.py:101: unsorted local import block.
• ty check src/ fails with 17 diagnostics, including nullable sid in src/benchflow/arena/agents_manifest.py, broad **kwargs constructor typing in src/benchflow/arena/bootstrap.py, and stale multiplayer= passed to run_floor_from_cli.
• uv run python -m benchflow.cli.main arena run --agents examples/casino/roster-smoke-oh.yaml --environment-manifest benchmarks/casinobench/environment.toml --out /tmp/benchflow-pr846-cli-smoke --max-turns 1 fails with No such option: --max-turns.
• The same arena run alias without --max-turns still crashes with TypeError: run_floor_from_cli() got an unexpected keyword argument 'multiplayer'.
• The documented examples/arena/README.md path is broken: eval run --agents examples/casino/agents.yaml ... fails before sandbox work because Roster.from_yaml rejects run-level keys (deadline_s, drive, idle_timeout_s, out, prompt, sandbox, services).
• On this arm64 Docker host, supported eval run --agents reaches image pull and fails for ghcr.io/yiminnn/casinobench-base:2.0.6: no matching manifest for linux/arm64/v8.

Strict review blockers:

• examples/casino/PROMPT.md is not used by examples/casino/run_floor.py; that script hardcodes PROMPT and sends that constant to execute_prompts.
• The arena/floor artifact surface still does not produce rollout-level result.json or results.jsonl; it writes roster.json, per-seat trajectories, and floor.json, which is below the current BenchFlow trajectory health contract.
• Generic arena core still contains casino-specific logic (/casino/observe?seat_id=... and casino nudge text).
• bootstrap_shared_env silently uses manifest.services[0], so multi-service manifests have no explicit service selection.
• Thermo scope remains too broad for this PR: 112 files / +7416 -55, including arena core, casino examples, medical benchmark/shim, docs, screenshots, and sample run artifacts.

Final verdict: blocked. Passing focused unit tests are not enough while ruff, ty, CLI, artifact contract, docs/example, and structural gates still fail.

[bingran-you]

Automation cleanup pushed: a5807d2d181a0e80ad2411dc52a75e285188c949 (fix(arena): wire floor run controls).

Fixed deterministic code/config blockers:

• Removed the stale deprecated arena run alias call path that passed multiplayer= into run_floor_from_cli().
• bench eval run --agents now forwards run-level controls into the native floor path: --reasoning-effort, --usage-tracking, and --agent-idle-timeout.
• FloorConfig now carries run-level reasoning/usage/idle-timeout config; per-seat reasoning_effort still overrides the run-level value.
• examples/casino/agents.yaml is now a pure roster accepted by Roster.from_yaml(); stale docs were updated to point at bench eval run --agents and the CLI reference documents the new floor flags.
• Fixed the nullable seat id in AgentsManifest.seats() by reusing the agent/model default id logic.
• Replaced broad sandbox constructor **kwargs with typed constructor calls in arena bootstrap.
• Cleared ruff failures, including the arena bootstrap import sort and remaining repo-wide ruff findings on the PR branch.

Validation run locally on the pushed tree:

• uv sync --extra dev --locked
• uv run ruff check .
• uv run ty check src
• uv run python -m pytest tests/test_cli_docs_drift.py::test_eval_run_flags_match_cli_md_bidirectional tests/test_arena_cli.py tests/test_concurrent_floor.py tests/test_agents_manifest.py tests/test_roster.py tests/test_arena_bootstrap.py tests/test_agent_driver.py -q -> 43 passed
• Roster.from_yaml("examples/casino/agents.yaml").seats() parses successfully.

Still blocked / no label changes:

• PR is still draft.
• GitHub reports mergeStateStatus=DIRTY / mergeable=CONFLICTING.
• gh pr checks 846 still reports no checks on the branch.
• The larger thermo-nuclear gates from prior reviews remain out of this deterministic cleanup pass: artifact contract / healthy trajectory proof, casino-specific logic in generic arena core, explicit service selection, medical implementation divergence, and overall broad scope/reviewability.

Keeping status:blocked + review:changes-requested until the draft/conflict/checks/artifact/structural gates are cleared.