Pattern observed
When debugging behavior driven by a visual or structured artifact (overlay PNG, generated SVG, log file, chart pixel data), agents tend to default to repeated reading + interpretation of the artifact. Each reading round consumes tokens AND can produce mutually inconsistent conclusions when the artifact is dense.
A targeted probe of the artifact's raw representation (pixel scan, JSON dump, regex extract) is faster, cheaper, and definitive.
Concrete example (me-fuji S149)
User reported "extracted curves are swapped" on a generated MTF overlay. Agent first round: study the overlay PNG, infer color/shape, build a hypothesis. Second round: study overlay again, revise hypothesis. Third round: ask user for clarification.
Replaced by:
```python
img = np.array(Image.open(chart_path).convert('RGB'))
for mm in range(0, 22):
x = plot_box.x_left + int(mm * plot_width / image_height_mm)
bands = scan_columns_for_orange_pixels(img, x)
print(f'mm={mm} bands_at_MTF={[mtf_of_band(b) for b in bands]}')
```
Two-line probe, ten seconds runtime. Settled the question definitively. The user was right; the overlay was misleading me.
Generalization
Source-of-truth artifacts in any project:
- Visual artifacts (images, plots, dashboards) — extract pixel data or underlying numbers
- Generated code/configs — diff against source, dump the AST, walk the tree
- API responses (logged) — dump raw JSON, query specific paths with jq
- Database state — write the SELECT, don't infer from app behavior
Suggested placement
`base/workflow/ai-workflow.md` Lessons Learned — alongside "Probe-first when issue body hypothesizes a mechanism" (already filed as #459).
Could be one combined section: "Probe the artifact, not your reading of it."
Related
- solid-ai-templates#459 (probe-first for issue-body mechanism hypotheses)
- solid-ai-templates#460 (disaggregate verdict / plausibility / accuracy)
- me-fuji dev journal §S149 (this lesson)
Pattern observed
When debugging behavior driven by a visual or structured artifact (overlay PNG, generated SVG, log file, chart pixel data), agents tend to default to repeated reading + interpretation of the artifact. Each reading round consumes tokens AND can produce mutually inconsistent conclusions when the artifact is dense.
A targeted probe of the artifact's raw representation (pixel scan, JSON dump, regex extract) is faster, cheaper, and definitive.
Concrete example (me-fuji S149)
User reported "extracted curves are swapped" on a generated MTF overlay. Agent first round: study the overlay PNG, infer color/shape, build a hypothesis. Second round: study overlay again, revise hypothesis. Third round: ask user for clarification.
Replaced by:
```python
img = np.array(Image.open(chart_path).convert('RGB'))
for mm in range(0, 22):
x = plot_box.x_left + int(mm * plot_width / image_height_mm)
bands = scan_columns_for_orange_pixels(img, x)
print(f'mm={mm} bands_at_MTF={[mtf_of_band(b) for b in bands]}')
```
Two-line probe, ten seconds runtime. Settled the question definitively. The user was right; the overlay was misleading me.
Generalization
Source-of-truth artifacts in any project:
Suggested placement
`base/workflow/ai-workflow.md` Lessons Learned — alongside "Probe-first when issue body hypothesizes a mechanism" (already filed as #459).
Could be one combined section: "Probe the artifact, not your reading of it."
Related