Remove confidence from PerceptionOutput and all downstream uses - #1
Open
trisha-ant wants to merge 1 commit into
Open
Remove confidence from PerceptionOutput and all downstream uses#1trisha-ant wants to merge 1 commit into
trisha-ant wants to merge 1 commit into
Conversation
VLM self-reported confidence is uncalibrated (0.867 when correct vs 0.857 when wrong) and was already optional in the harness. This drops the field entirely so future work derives reliability from session history rather than a noisy scalar. - PerceptionOutput: drop confidence (both _base.py and types.py) - response_to_output / build_history_text: stop parsing/formatting it - run.py: drop confidence from history dicts, prediction records, mean_confidence metric, and results table - benchmark/metrics.py: remove calibration block (mean_confidence, ECE, calibration_bins, confidence_when_correct/wrong) - All variants (perception/ + experiments/prompt/): drop confidence from JSON output schema and stop reading it from history - duration_aware: remove _confidence_gate; variant now relies on duration-aware prompt anchoring alone - Docs: update program.md, experiments/prompt/README.md
This was referenced Apr 17, 2026
trisha-ant
marked this pull request as ready for review
April 23, 2026 15:53
This was referenced Apr 28, 2026
trisha-ant
added a commit
that referenced
this pull request
Apr 30, 2026
…olds up) Master config (opus-4-6, temperature=0, no thinking, anchored prompts): exact: 81.7 +/- 2.4 (runs 82.8, 83.3, 79.0) 1.5fold: 65.9 +/- 0.0 (deterministic -- temp=0 + strong anchoring) 2fold: 68.4 +/- 9.8 pretzel: 92.5 +/- 8.5 adj: 100.0 Only diff from master's hybrid.py is confidence field removed from JSON schema (PR #1). Master's 82.8 was a typical draw. vs vote3_mm@4.7 (80.8+/-0.5): +0.9pp, t~0.61 -- not significant. Statistically equivalent overall. Per-stage trade-off: hybrid@4.6 wins 1.5fold (66 vs 45) and pretzel (93 vs 85) vote3_mm@4.7 wins 2fold (98 vs 68) and variance (0.5 vs 2.4) _base.py reverted to 4.7 config; hybrid46 JSONs archived.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
VLM self-reported confidence is uncalibrated noise — across saved runs, mean confidence is 0.867 when correct vs 0.857 when wrong, so it carries essentially no signal. The harness already derives reliability from session history (stability, temporal consistency) rather than this scalar, and
d9b89f2had already made it optional. This finishes the removal so future variants and the upcoming z-slice subagent work don't have to thread a dead field through every prompt and output.What
PerceptionOutput: dropconfidencefield (bothperception/_base.pyandgently_perception/types.py)response_to_output/build_history_text: stop parsing/formatting itrun.py: drop confidence from history dicts, prediction records,mean_confidence, and the results tablebenchmark/metrics.py: remove the calibration block (ECE, bins, conf-when-correct/wrong)"confidence"from the JSON output schema and stop reading it from historyduration_aware: remove_confidence_gate(it gated transitions on confidence); the variant now relies on duration-aware prompt anchoring aloneprogram.md,experiments/program.md,experiments/prompt/README.md36 files, +44 / −323.
Behavioral note
duration_aware's previous benchmark numbers (81.3%) included the confidence gate; they no longer apply and would need a re-run.Verification
perception.*,experiments.prompt.*,gently_perception,benchmark.metrics,run) import cleanlyPerceptionOutputdefinitions rejectconfidence=withTypeErrorresponse_to_output/build_history_textwork and silently ignore legacyconfidencekeys in input JSON / history dictspython run.py --helpexits 0grep -rn '\.confidence' --include='*.py'→ emptycompute_metrics+format_metrics_summaryrun end-to-end on a mock report