result.json schema bloat: top-level/agent_result duplication and started_at/finished_at not ISO 8601

[bingran-you]

Summary

Two related schema-hygiene issues in result.json that compound:

A. Field duplication between top-level and agent_result

result.json writes three fields twice — once at top level and once inside the nested agent_result block:

{
  "n_tool_calls": 9,                  ←
  "n_skill_invocations": 0,           ← duplicated
  "n_prompts": 1,                     ←
  "agent_result": {
    "n_tool_calls": 9,                ← duplicated
    "n_skill_invocations": 0,         ← duplicated
    "n_prompts": 1                    ← duplicated
  }
}

Source: src/benchflow/rollout.py:514-520. Consumers must pick one location; whichever they pick goes stale on the next refactor.

B. started_at / finished_at are not ISO 8601

{
  "started_at": "2026-05-24 21:48:26.264901",      ← str(datetime), space sep, no TZ
  "finished_at": "2026-05-24 21:50:04.087940",     ← same
  "idle_timeout_info": {
    "last_activity_at": "2026-05-25T04:51:45.254841+00:00"   ← ISO 8601 + TZ
  }
}

result.json mixes formats within the same blob. The top-level fields come from rollout.py:540-541 (str(result.started_at)) but rewards.jsonl (rollout.py:262, 282) uses .isoformat() — same project, two parse paths.

Repro

J=$(ls -d /tmp/v05-jobs/test-include/20*/weighted-gdp-calc__*/ | head -1)

# A: duplication
jq '{
  top_calls: .n_tool_calls,
  nested_calls: .agent_result.n_tool_calls,
  top_prompts: .n_prompts,
  nested_prompts: .agent_result.n_prompts
}' "$J/result.json"

# B: timestamp drift
jq -r '.started_at, .finished_at' "$J/result.json"          # space-separated
jq -r '.ts' "$J/rewards.jsonl"                              # ISO 8601
jq -r '.idle_timeout_info.last_activity_at' "$J/result.json" # ISO 8601 + TZ

Severity

P2.

• Duplication is bloat + latent staleness risk; not a current bug, but a refactor footgun.
• Timestamp drift forces every consumer (dashboard, check_results, downstream analysis) to branch on format. Already burned #491 (dashboard advisory drift) territory.

Proposed fix

A

Pick one canonical location. The flat top-level fields are likely the older shape; agent_result is the typed-model form. Either:

• Keep agent_result and remove the top-level duplicates (preferred — agent_result is the typed shape that downstream consumers should use).
• Or keep flat top-level and remove from agent_result.

Either way, this is a one-breaking-change bullet.

B

Replace str(result.started_at) with result.started_at.isoformat() (TZ-aware datetime needs astimezone(UTC).isoformat() if it isn't already). Audit rollout.py for other str(some_datetime) calls — a quick grep:

grep -n "str(.*started_at\|str(.*finished_at\|str(.*at\b" src/benchflow/rollout.py

Add a unit test that asserts every timestamp emitted by rollout.py parses with datetime.fromisoformat() and is TZ-aware.

Found via

v0.5 e2e validation §2 (threejs-to-obj) and §3 (data-to-d3) at 5a7a326.

[bingran-you]

Daily scan hygiene (2026-07-08): still open and still repo-owned; no open PR currently targets this result-schema cleanup.

Read-only scan confirms this remains a schema/rollout hygiene issue rather than an off-repo task problem. Next safe step is a focused schema-decision PR: choose the canonical metrics location (agent_result vs top-level duplicates) and normalize started_at / finished_at to timezone-aware ISO 8601. Keeping enhancement, P2, status:ready, area:rollout.

[mychiffonn]

I can take a look at this and finish PR this week. If you have an internal plan let me know.

[bingran-you]

Daily scan response (2026-07-09): yes, this is still the active internal plan for #528.

Recommended implementation plan:

1. Treat agent_result as the canonical nested result object and keep top-level duplicates only through a compatibility path if removing them would break current consumers.
2. Normalize started_at / finished_at to timezone-aware ISO 8601 at the rollout-result serialization boundary, not in individual downstream exporters.
3. Add a regression test that builds a representative result.json and asserts the stable schema shape plus timezone-aware timestamps.
4. If the patch removes any top-level duplicate fields, call that out explicitly in the PR body as a result-schema compatibility change.

Labels are now status:ready + help wanted; please link the PR here when you start so the project board can move to In Progress.