fix(learner): eliminate false-positive proposals (structural error detection + by-design file exclusions + trigram filter)#29
Open
juanparisma wants to merge 1 commit into
Conversation
…etection, by-design file exclusions, workflow trigram filter One real session produced 190 proposals of which 189 were junk. Three root causes, all fixed: 1. observe_v3.py flagged is_error=True whenever the output text contained "error"/"failed" keywords — including inside the CONTENT of successful Reads (e.g. source code with 'throw new Error'). Proof: err_msg was identical to sample_output on the junk proposals. Now detection is (a) structural harness failure shapes for every tool (dict error field, tool_use_error marker, start-of-output Error/InputValidationError) and (b) hard failure markers (Permission denied, command not found, Traceback, npm ERR!, ...) only for Bash, whose output is a run result rather than arbitrary file content. 2. user_correction counted any file edited 3+ times, but journals and control panels (hot.md, brief.md, working-memory.md, MEMORY.md, CLAUDE.md, daily summaries, wiki dirs) are re-edited BY DESIGN during end-of-day flows. They are now excluded via BYDESIGN_RE. 3. workflow_chain proposed every tool trigram repeated 2+ times, which is dominated by generic coding activity (Bash>Bash>Bash x380 in one session, 154 junk proposals). Trigrams now require 5+ repeats, >= 2 distinct tools, and at least one non-generic (MCP/custom) tool. The learner also re-validates the legacy is_error flag against hard markers (isRealError) because observations written by older observers remain contaminated, and PATTERN 5 (agent errors) uses the same validation instead of keyword sniffing. A/B on a real 8000-line observation window: 134 proposals with the old code -> 17 with this fix, and the 17 survivors are all legitimate signals (3 genuine tool failures, 7 genuinely iterated source files, 7 repeated MCP workflow sequences). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problema
En una sesión real del 2026-07-16, el session-learner generó 190 proposals de las cuales 189 eran basura. Tres causas raíz, verificadas contra las observaciones reales:
error_resolutionfantasma —observe_v3.pymarcabais_error=Truesi el output contenía la subcadenaerror/faileden cualquier parte, incluido el contenido de archivos leídos con éxito (p. ej. código fuente conthrow new Error). Prueba: en las proposals falsas,err_msgera idéntico asample_outputy no contenía ningún error.user_correctionfalsas — contaba como corrección cualquier archivo editado 3+ veces, pero los diarios/paneles de control (hot.md,brief.md,working-memory.md, daily summaries, wikis) se re-editan por diseño en los flujos de cierre (/eod). Una bitácora con 11 ediciones ≠ 10 correcciones del usuario.workflow_chainruido — proponía cualquier trigrama de tools repetido 2+ veces; eso lo domina la actividad genérica de programar (Bash>Bash>Bash×380 en una sesión → 154 proposals inútiles).Fix
error, marcadortool_use_error, output que EMPIEZA conError/InputValidationError) + (b) marcadores duros (Permission denied,command not found,Traceback,npm ERR!…) solo para Bash, cuyo output es resultado de ejecución y no contenido arbitrario.isRealError(): re-valida el flagis_errorcontra marcadores duros — necesario porque las observaciones ya escritas por observers viejos siguen contaminadas y el learner relee esa ventana.user_correction: excluye archivos re-editados por diseño (BYDESIGN_RE).workflow_chain: exige 5+ repeticiones, ≥2 tools distintas y ≥1 tool no-genérica (MCP/custom).isRealErroren vez de sniffing de keywords.Verificación
throw new Error, Bash exitoso, Grep sobre logs con "error") danFalse; los 3 errores reales (Permission denied,String to replace not found,File does not exist) danTrue.Corriendo en producción en mis 2 PCs desde hoy sin regresiones.
🤖 Generated with Claude Code