Tracewell

An agent that shows its work — and a harness that scores it.

Tracewell is a small research-agent project built around one question most agent demos avoid: when the agent fails, why did it fail, and how would you know before a user did?

Most agents are built to complete a task and stop there. Tracewell is built to complete a task and produce a full, human-readable trace of every decision it made — which is then run against a fixed evaluation harness that scores it, not just as pass/fail, but by failure category. The goal isn't a flashy agent. It's a small, rigorous example of the full loop: build → trace → evaluate → diagnose → fix → re-measure.

---

The problem

Agents fail silently. They pick the wrong tool, hallucinate an intermediate fact, loop unnecessarily, or produce a confident, plausible, wrong answer — and from the outside, a wrong answer looks identical to a right one. Most agent projects have no visibility into why the agent did what it did, and no standard way to say "this agent is X% reliable at this task, and here's the failure mode costing the most."

Tracewell is an attempt to fix both problems in one small repo: full decision traceability, and a harness that turns failures into a measurable, categorized signal.

---

How it works

                ┌─────────────┐
   task ───────▶│   Planner   │◀────────────┐
                └──────┬──────┘              │
                       │ decides next action  │
                       ▼                      │
                ┌─────────────┐               │
                │  Tool node  │───────────────┘
                │(search/calc)│   loop until enough
                └─────────────┘   facts gathered
                       │
                       ▼
                ┌─────────────┐
                │   Writer    │──▶ final answer
                └─────────────┘    (only from state,
                                     never untraced memory)

Every planner decision, tool call, and state transition is logged to a structured trace — see examples/sample_trace.json for a real, annotated example.

---

Example trace

{
  "step": 3,
  "action": "web_search",
  "input": "population of Lagos 2026",
  "reasoning": "need current population figure to support claim 2",
  "output_summary": "found result on worldometers.info",
  "timestamp": "..."
}

Every claim in the final output can be traced back through the log to the exact tool call and reasoning step that produced it.

---

The evaluation harness

Tracewell is tested against a fixed set of 10-15 tasks (not re-generated per run, so results are comparable over time), scored not just pass/fail but by failure category:

Category	Meaning
hallucinated_source	claim isn't actually supported by the cited source
wrong_tool	wrong tool chosen for the step (e.g. search instead of calculator)
incomplete	agent stopped before gathering required facts
loop_exceeded	hit the iteration cap without finishing
pass	met all task criteria

Running harness/run_eval.py produces a report like:

9/14 passed (64%)
Failures: hallucinated_source (3), wrong_tool (1), incomplete (1)
Dominant failure mode: hallucinated_source

Results

Before fix:

Metric	Value
Pass rate	TBD
Dominant failure mode	TBD

After fix: (if applicable)

Metric	Value
Pass rate	TBD
What changed	TBD

---

What I learned

---

Running it locally

git clone https://github.com/<your-username>/tracewell.git
cd tracewell
pip install -r requirements.txt

# Run the agent on a single task
python -m agent.graph --task "your task here"

# Run the full evaluation harness
python -m harness.run_eval

Requires an API key for your chosen LLM provider and search tool — see .env.example.

---

Project structure

tracewell/
├── agent/          # LangGraph agent — planner, tools, state
├── harness/         # Fixed task set, scorer, eval runner
├── traces/          # Per-run trace logs
├── results/          # Generated eval reports
└── examples/          # Annotated sample trace

---

Why this exists

Built as a small, honest artifact on agent reliability — not a bigger agent, a more measured one. Feedback and issues welcome.