Skip to content

evaluation_system.md

Latest commit

 

History

History
251 lines (174 loc) · 11.8 KB

evaluation_system.md

File metadata and controls

251 lines (174 loc) · 11.8 KB
Raw

Evaluation System

This doc describes how workflow evaluation works now.

The current system is file-backed and execution-oriented:

  • the evaluation plan lives in evaluation/evaluation_plan.json
  • evaluation step config lives in evaluation/step_config.json
  • evaluation execution runs in an internal sandbox under evaluation/runs/iteration-0[/group]
  • the final report is published to evaluation/runs/{targetRunFolder}/evaluation_report.json

What Evaluation Is For

Evaluation is the workflow's goal-measurement layer for completed execution runs.

It answers:

  • did a target run actually satisfy the success criteria in soul/soul.md?
  • what did each evaluation step conclude, with what evidence?

It is separate from:

  • execution-time pre-validation (mechanical run-shape checks)
  • Pulse per-run triage (operational breakage: errors, skipped steps, empty artifacts, hallucinated successes)
  • step learning
  • workflow execution itself

Division of labor: Pulse owns "did it run right"; evals own "did it achieve the goal." Eval steps should map one-to-one to success criteria and never duplicate operational checks — see the evaluation-plan guidance template for the authoring contract. Pulse reads eval reports and records evidence-stamped goal progress in builder/improve.html Reflection history. The durable goal definition remains soul/soul.md; there is no separate numeric metrics layer or duplicate Goal/Profile card.

Current Mental Model

Use this mental model:

  1. Build or edit the eval plan in evaluation/evaluation_plan.json.
  2. Point evaluation at a completed execution run folder such as iteration-8/production.
  3. Evaluation steps execute in an internal eval sandbox.
  4. Those eval steps read the original execution artifacts through {{TARGET_RUN_PATH}}.
  5. A scoring phase produces evaluation_report.json.
  6. The report is published back under the requested target run folder path inside evaluation/runs/.

Eval Plan Files

The core evaluation files are:

  • evaluation/evaluation_plan.json
  • evaluation/step_config.json
  • evaluation/eval_layout.json
  • learnings/ — shared with execution-plan learnings (see "Learnings And Step Config In Eval Mode" below)

Frontend eval mode loads these files directly. The eval plan is not embedded in the main workflow manifest structure.

Eval steps can be scoped to the route or artifact they apply to:

{
  "id": "eval-bid-submission",
  "title": "Evaluate Bid Submission",
  "description": "Verify bid submission artifacts...",
  "applies_to_routes": [
    { "routing_step_id": "workflow-mode-router", "route_ids": ["route-bid"] }
  ]
}

When this field is present, the runtime checks the target run's routing-evaluation.json before launching the eval step. Non-applicable checks are skipped and recorded in the final report with max_score: 0, so a run is not penalized for route paths it did not take.

Current frontend behavior is implemented in useEvaluationPlanData.ts.

Eval Mode In The UI

The UI still has a separate eval mode.

Current behavior:

  • plan mode shows the main workflow
  • eval mode shows evaluation/evaluation_plan.json
  • eval steps use evaluation/step_config.json
  • eval layout is separate from the main workflow layout

So evaluation is still a first-class workspace view, not just a hidden background feature.

Running Evaluation

The runtime entry point is ExecuteEvaluationOnly(...).

Current behavior:

  • reads evaluation/evaluation_plan.json
  • requires a target run folder
  • computes an internal eval run folder using the workshop-style iteration-0 sandbox
  • sets isEvaluationMode = true
  • injects TARGET_RUN_PATH
  • applies evaluation/step_config.json
  • runs the evaluation steps
  • runs a final scoring phase

This is implemented in evaluation_execution.go.

Internal Eval Sandbox

One of the biggest architecture changes is the eval run folder model.

Evaluation does not primarily execute inside:

  • evaluation/runs/{targetRunFolder}/...

Instead it executes inside an internal sandbox:

  • evaluation/runs/iteration-0
  • or evaluation/runs/iteration-0/<group>

The target run folder is still important, but it is the thing being evaluated, not the main place eval steps execute.

This is why the code uses:

  • internal eval run folder for execution
  • published target run folder for the final report

TARGET_RUN_PATH

TARGET_RUN_PATH is the main bridge between evaluation and the original workflow run.

Current behavior:

  • the runtime injects TARGET_RUN_PATH as the absolute path to the original execution folder
  • eval steps are expected to read the original run artifacts through that variable
  • execution artifacts normally remain inside their producer step folder; they are not copied to the execution root

For example, the original run might be:

  • runs/iteration-12/production/execution

The eval step runs in the eval sandbox, but reads the original artifacts through:

  • {{TARGET_RUN_PATH}}

This is the correct way to reference original execution outputs in eval steps.

Eval descriptions must name the producer-relative artifact path, for example:

  • {{TARGET_RUN_PATH}}/score-and-plan/trade_plan_summary.json

When a generated sub-step folder makes the exact name variable, declare a narrow single-match pattern under the current TARGET_RUN_PATH. Do not use ambiguous wording such as "the file under TARGET_RUN_PATH," because it can be misread as requiring a direct child of the execution root.

Report Phase

After eval steps finish, the runtime builds a single evaluation_report.json covering every eval step — directly in Go (runEvaluationReportPhase in evaluation_execution.go). There is no scoring agent, no combined LLM scoring call, and no learn_code scoring fast path (all removed); each eval step's own structured output is its verdict.

Report shape

The on-disk report contains:

  • target_run_folder, generated_at
  • step_scores[], one entry per eval step:
    • step_id
    • placeholder score: 0 / max_score: 0 with fixed reasoning/evidence text pointing readers at output_content ("Final scoring is disabled; this report preserves the eval step output for review")
    • output_content — the eval step's own structured output, attached from the first of: output_content.json, the step's context_output file, a pre_validation file, or context_output.json in the eval sandbox (enrichEvaluationReportWithStepOutputs)
  • eval steps skipped by applies_to_routes appear as skipped entries with max_score: 0, so a run is not penalized for route paths it did not take

output_content is the source of truth for each eval step's verdict: the step should emit its own score, max_score, reasoning, and evidence (plus any domain-specific judgment dimensions) as structured output, enforced by the step's validation schema. Consumers — Pulse, Goal Advisor, report review, and the UI — read the per-step verdicts; nothing numeric is aggregated downstream.

Where the report lands

Whichever path produces it, the runtime writes the report to:

  • internal path: evaluation/runs/{internalEvalRunFolder}/evaluation_report.json
  • published path: evaluation/runs/{targetRunFolder}/evaluation_report.json

The publish step is what makes evaluation reports line up with the execution run the user asked about.

Auto-Evaluation

Evaluation can also run automatically after workflow execution.

Current behavior:

  • after a successful batch group execution, the workflow checks whether evaluation/evaluation_plan.json exists
  • if it exists, auto-evaluation runs for that target run folder
  • evaluation failures do not fail the original group execution

This behavior is implemented in:

Evaluation Costs

The old mental model of evaluation writing a standalone evaluation/runs/.../token_usage.json as the main source of truth is no longer the right model.

Current cost architecture:

  • evaluation cost data is stored in the costs/ ledger under the evaluation scope
  • /api/workflow/costs returns eval cost data as evaluation_token_usage
  • the UI merges execution and evaluation costs when showing workflow run cost summaries

So:

  • execution costs -> costs/execution/...
  • evaluation costs -> costs/evaluation/...

For the full cost architecture, see cost_and_log_measurement.md.

Evaluation Reports API

The evaluation report UI is driven by /api/workflow/evaluation-reports.

Current behavior:

  • scans evaluation/runs/*
  • finds evaluation_report.json
  • supports both bare iteration folders and nested group folders
  • returns aggregate statistics across reports
  • can filter to a specific run folder

This is implemented in workflow.go.

Learnings And Step Config In Eval Mode

Evaluation mode has its own step config but shares the learnings namespace with execution steps.

Current behavior:

  • step config comes from evaluation/step_config.json (separate from planning/step_config.json)
  • learnings go under learnings/{stepID}/ — the same folder execution steps write to
  • cross-plan step-ID uniqueness between planning/plan.json and evaluation/evaluation_plan.json is enforced at write time by validateCrossPlanStepIDUniqueness, so the shared namespace cannot collide

Eval steps therefore reuse workflow learnings (e.g. a _global SKILL.md written by execution steps is visible to eval steps), but keep an independent step_config.json.

Validation Of The Eval Plan

The system includes validate_evaluation_plan for checking the file after edits.

Current validation checks include:

  • JSON parse validity
  • non-empty step list
  • unique IDs
  • required title and description
  • validation of pre_validation regex and JSONPath rules where present

This is implemented in evaluation_helpers.go.

What Changed From Older Docs

The older evaluation doc described a different architecture in a few places.

Current corrections:

  • evaluation is file-backed, not centered on a separate designer manager architecture
  • eval execution uses an internal evaluation/runs/iteration-0[/group] sandbox
  • the published report is copied to the target run folder path
  • evaluation costs come from the costs/ ledger, not primarily from legacy per-run token files
  • the eval plan is edited and loaded directly from evaluation/evaluation_plan.json
  • the combined LLM scoring agent and the __evaluation_scoring__ learn_code fast path were removed — the report is assembled in Go from per-step output_content
  • the numeric metrics layer (planning/metrics.json, db/metrics_history.jsonl, propose_metric/retire_metric) was removed (2026-07-01) — soul/soul.md defines the goal and Pulse records evidence-stamped progress over time in builder/improve.html

Practical Summary

Use this mental model:

  • edit the eval plan in evaluation/evaluation_plan.json
  • run evaluation against a completed execution run
  • eval steps execute in the internal eval sandbox
  • eval steps read original execution artifacts via {{TARGET_RUN_PATH}}
  • scoring publishes evaluation_report.json back to the requested target run folder
  • costs for eval runs are tracked in the costs/evaluation/ ledger

Related Docs