From 5b0cd84801707c1334dcad34ae8bba44162f36f2 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:09:44 -0400 Subject: [PATCH 1/7] =?UTF-8?q?=F0=9F=94=AC=20Define=20impact-based=20seve?= =?UTF-8?q?rity=20levels=20in=20review=20prompts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bare "high/medium/low" labels give agents no shared calibration standard, leading to inconsistent severity across reviews. Defining each level in terms of real-world impact (data loss, exploitability, blast radius) aligns all agents on the same scale and naturally prevents low-value findings from being over-rated. Inspired by the impact × breadth scoring pattern from research-oriented analysis skills. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index 69671b00..327c4474 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -42,7 +42,10 @@ After reviewing, provide: 1. A brief summary of what the commit does 2. Any issues found, listed with: - - Severity (high/medium/low) + - Severity, using these definitions: + - **high**: Will cause data loss, security breach, crash, or incorrect results in production + - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability + - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - A brief explanation of the problem and suggested fix @@ -61,7 +64,10 @@ After reviewing, provide: 1. A brief summary of what the changes do 2. Any issues found, listed with: - - Severity (high/medium/low) + - Severity, using these definitions: + - **high**: Will cause data loss, security breach, crash, or incorrect results in production + - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability + - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - A brief explanation of the problem and suggested fix @@ -82,7 +88,10 @@ After reviewing, provide: 1. A brief summary of what the commits do 2. Any issues found, listed with: - - Severity (high/medium/low) + - Severity, using these definitions: + - **high**: Will cause data loss, security breach, crash, or incorrect results in production + - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability + - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - A brief explanation of the problem and suggested fix @@ -867,7 +876,11 @@ const SystemPromptSecurity = `You are a security code reviewer. Analyze the code 10. **Error handling**: Information leakage via error messages, missing error checks on security-critical operations For each finding, provide: -- Severity (critical/high/medium/low) +- Severity, using these definitions: + - **critical**: Actively exploitable vulnerability allowing remote code execution, auth bypass, or data exfiltration + - **high**: Exploitable vulnerability requiring specific conditions or limited attacker capability + - **medium**: Weakness that increases attack surface or could become exploitable with other changes + - **low**: Defense-in-depth improvement or theoretical concern with no practical exploit path in current code - File and line reference - Description of the vulnerability - Suggested remediation From cc39ed8e142396654a1f660e90bf366388fb13f1 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:11:18 -0400 Subject: [PATCH 2/7] =?UTF-8?q?=F0=9F=94=AC=20Require=20concrete=20harm=20?= =?UTF-8?q?articulation=20in=20review=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace "brief explanation of the problem" with "what specifically goes wrong if this is not fixed." This is the articulation test pattern from research-oriented analysis skills — every finding must justify itself with concrete impact reasoning, not just pattern-matching against a checklist. Findings like "this violates best practices" become impossible to write when the prompt demands specific harm. This is the single most effective noise reduction technique across the mop-mapping skill set. Applied to all review types: standard, dirty, range, security, and design. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index 327c4474..23855108 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -47,7 +47,8 @@ After reviewing, provide: - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - - A brief explanation of the problem and suggested fix + - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") + - Suggested fix If you find no issues, state "No issues found." after the summary.` @@ -69,7 +70,8 @@ After reviewing, provide: - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - - A brief explanation of the problem and suggested fix + - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") + - Suggested fix If you find no issues, state "No issues found." after the summary.` @@ -93,7 +95,8 @@ After reviewing, provide: - **medium**: Will cause degraded behavior under specific conditions, or blocks future maintainability - **low**: Minor improvement opportunity with no immediate functional impact - File and line reference where possible - - A brief explanation of the problem and suggested fix + - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") + - Suggested fix If you find no issues, state "No issues found." after the summary.` @@ -846,10 +849,12 @@ After reviewing, provide: 1. A brief summary of what the design proposes 2. PRD findings, listed with: - Severity (high/medium/low) - - A brief explanation of the issue and suggested improvement + - What specifically goes wrong during implementation if this gap is not addressed + - Suggested improvement 3. Task list findings, listed with: - Severity (high/medium/low) - - A brief explanation of the issue and suggested improvement + - What specifically goes wrong during implementation if this gap is not addressed + - Suggested improvement 4. Any missing considerations not covered by the design 5. A verdict: Pass or Fail with brief justification @@ -882,7 +887,7 @@ For each finding, provide: - **medium**: Weakness that increases attack surface or could become exploitable with other changes - **low**: Defense-in-depth improvement or theoretical concern with no practical exploit path in current code - File and line reference -- Description of the vulnerability +- The specific code path an attacker would exploit and what they gain - Suggested remediation If you find no security issues, state "No issues found." after the summary. From 9150177fb00749fb6790116639f16fb3159d858e Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:13:02 -0400 Subject: [PATCH 3/7] =?UTF-8?q?=F0=9F=94=AC=20Add=20evidence=20thresholds?= =?UTF-8?q?=20to=20suppress=20speculative=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /rethink skill uses explicit evidence thresholds — "1 observation is a data point, 3+ is a pattern worth investigating." The /verify skill grounds every check in specific data. Applied here as negative prompt instructions that suppress the most common false positive categories: hypothetical issues in unseen code, style preferences, unfounded "missing tests" claims, and flagging patterns that match existing codebase conventions. Security reviews get a lighter version — they should still err toward reporting, but not flag theoretical vulnerabilities in untouched code. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index 23855108..ccab9fb1 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -36,7 +36,11 @@ const SystemPromptSingle = `You are a code reviewer. Review the git commit shown 4. **Regressions**: Changes that might break existing functionality 5. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming -Do not review the commit message itself - focus only on the code changes in the diff. +Do not report issues without specific evidence in the diff. In particular, do not report: +- Hypothetical issues in code not shown in the diff +- Style preferences or naming opinions that do not affect correctness +- "Missing tests" unless the change introduces testable behavior with no coverage +- Patterns that are consistent with the codebase conventions visible in context After reviewing, provide: @@ -61,6 +65,12 @@ const SystemPromptDirty = `You are a code reviewer. Review the following uncommi 4. **Regressions**: Changes that might break existing functionality 5. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming +Do not report issues without specific evidence in the diff. In particular, do not report: +- Hypothetical issues in code not shown in the diff +- Style preferences or naming opinions that do not affect correctness +- "Missing tests" unless the change introduces testable behavior with no coverage +- Patterns that are consistent with the codebase conventions visible in context + After reviewing, provide: 1. A brief summary of what the changes do @@ -84,7 +94,11 @@ const SystemPromptRange = `You are a code reviewer. Review the git commit range 4. **Regressions**: Changes that might break existing functionality 5. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming -Do not review the commit message itself - focus only on the code changes in the diff. +Do not report issues without specific evidence in the diff. In particular, do not report: +- Hypothetical issues in code not shown in the diff +- Style preferences or naming opinions that do not affect correctness +- "Missing tests" unless the change introduces testable behavior with no coverage +- Patterns that are consistent with the codebase conventions visible in context After reviewing, provide: @@ -880,6 +894,10 @@ const SystemPromptSecurity = `You are a security code reviewer. Analyze the code 9. **Concurrency issues**: Race conditions leading to security bypasses, TOCTOU vulnerabilities 10. **Error handling**: Information leakage via error messages, missing error checks on security-critical operations +Only report vulnerabilities with a plausible exploit path visible in the diff. Do not report: +- Theoretical vulnerabilities in code not touched by this change +- Generic hardening suggestions unrelated to the specific code under review + For each finding, provide: - Severity, using these definitions: - **critical**: Actively exploitable vulnerability allowing remote code execution, auth bypass, or data exfiltration From 02d30336a6303255eb9e7687f08505b1f45fe232 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:14:37 -0400 Subject: [PATCH 4/7] =?UTF-8?q?=F0=9F=94=AC=20Add=20intent-implementation?= =?UTF-8?q?=20alignment=20check=20(cold-read=20prediction)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /verify skill's "recite" phase is its most powerful technique: read only the title, predict what the content should be, then check alignment. Applied here by reversing the old instruction "Do not review the commit message" — the commit message now becomes the primary lens for evaluating the diff. When a commit says "fix race condition" but the diff adds a mutex on the wrong resource, that's a high-value finding that pure diff-scanning misses. Intent-implementation gaps are now the first check category, above bugs and security, because they catch the class of errors where the code is internally consistent but doesn't do what the developer intended. The dirty-changes prompt is unchanged since uncommitted changes have no commit message to analyze. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index ccab9fb1..0cd43827 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -28,13 +28,18 @@ Return only the final review content. Do NOT include process narration, progress If you use tools while reviewing, finish all tool use before emitting the final review, and put the final review only after the last tool call.` // SystemPromptSingle is the base instruction for single commit reviews -const SystemPromptSingle = `You are a code reviewer. Review the git commit shown below for: +const SystemPromptSingle = `You are a code reviewer. Review the git commit shown below. -1. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions -2. **Security**: Injection vulnerabilities, auth issues, data exposure -3. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps -4. **Regressions**: Changes that might break existing functionality -5. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming +First, read the commit message to understand the developer's intent. Then read the diff and check whether the code changes fully and correctly achieve that intent. Gaps between stated intent and actual implementation are high-value findings. + +Check for: + +1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit message claims? +2. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions +3. **Security**: Injection vulnerabilities, auth issues, data exposure +4. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps +5. **Regressions**: Changes that might break existing functionality +6. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming Do not report issues without specific evidence in the diff. In particular, do not report: - Hypothetical issues in code not shown in the diff @@ -86,13 +91,18 @@ After reviewing, provide: If you find no issues, state "No issues found." after the summary.` // SystemPromptRange is the base instruction for commit range reviews -const SystemPromptRange = `You are a code reviewer. Review the git commit range shown below for: +const SystemPromptRange = `You are a code reviewer. Review the git commit range shown below. -1. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions -2. **Security**: Injection vulnerabilities, auth issues, data exposure -3. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps -4. **Regressions**: Changes that might break existing functionality -5. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming +First, read the commit messages to understand the developers' intent. Then read the diff and check whether the code changes fully and correctly achieve that intent. Gaps between stated intent and actual implementation are high-value findings. + +Check for: + +1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit messages claim? +2. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions +3. **Security**: Injection vulnerabilities, auth issues, data exposure +4. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps +5. **Regressions**: Changes that might break existing functionality +6. **Code quality**: Duplication that should be refactored, overly complex logic, unclear naming Do not report issues without specific evidence in the diff. In particular, do not report: - Hypothetical issues in code not shown in the diff From 359d576af734d8993c16ee2f9b6255d8231c7729 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:16:38 -0400 Subject: [PATCH 5/7] =?UTF-8?q?=F0=9F=94=AC=20Add=20self-review=20quality?= =?UTF-8?q?=20gate=20before=20output?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /verify and /synthesize skills both enforce quality gates — checks that must pass before output is considered complete. Applied here as a final self-verification instruction: every finding must reference a specific diff location, severity must match the described impact, and no two findings may contradict each other. Findings that fail these checks are dropped. This catches the most embarrassing review failures (high-severity verdict with no actual line references, "pass" with critical findings listed) at near-zero cost since the model performs the check during the same generation. Applied to all review types: standard, dirty, range, and security. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index 0cd43827..24f0147a 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -59,6 +59,8 @@ After reviewing, provide: - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") - Suggested fix +Before finalizing, verify your review: every finding must reference a specific file and line number, the severity must match the impact you described, and no two findings should contradict each other. Drop any finding that fails these checks. + If you find no issues, state "No issues found." after the summary.` // SystemPromptDirty is the base instruction for reviewing uncommitted (dirty) changes @@ -88,6 +90,8 @@ After reviewing, provide: - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") - Suggested fix +Before finalizing, verify your review: every finding must reference a specific file and line number, the severity must match the impact you described, and no two findings should contradict each other. Drop any finding that fails these checks. + If you find no issues, state "No issues found." after the summary.` // SystemPromptRange is the base instruction for commit range reviews @@ -122,6 +126,8 @@ After reviewing, provide: - What specifically goes wrong if this is not fixed (concrete harm, not "violates best practices") - Suggested fix +Before finalizing, verify your review: every finding must reference a specific file and line number, the severity must match the impact you described, and no two findings should contradict each other. Drop any finding that fails these checks. + If you find no issues, state "No issues found." after the summary.` // PreviousReviewsHeader introduces the previous reviews section @@ -918,6 +924,8 @@ For each finding, provide: - The specific code path an attacker would exploit and what they gain - Suggested remediation +Before finalizing, verify your review: every finding must reference a specific file and line number and describe a plausible exploit path. The severity must match the exploitability you described. Drop any finding that fails these checks. + If you find no security issues, state "No issues found." after the summary. Do not report code quality or style issues unless they have security implications.` From 39345354f6dce7ea959642639661e869f0671f25 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Tue, 24 Mar 2026 13:19:11 -0400 Subject: [PATCH 6/7] =?UTF-8?q?=F0=9F=94=AC=20Add=20evidence=20thresholds?= =?UTF-8?q?=20to=20insights=20analysis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /rethink skill's evidence accumulation pattern — "1 observation is a data point, 3+ is a pattern worth investigating" — directly applies to the insights system. Without explicit thresholds, the insights agent may recommend guideline changes from 1-2 occurrences (noise) or hesitate on strong 6+ patterns. Added tiered thresholds to the recurring patterns section and gated guideline suggestions on minimum 3 occurrences. This helps close the feedback loop between review noise and guideline refinement with appropriate confidence levels. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/insights.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/internal/prompt/insights.go b/internal/prompt/insights.go index d36a08e0..5353eea1 100644 --- a/internal/prompt/insights.go +++ b/internal/prompt/insights.go @@ -19,7 +19,12 @@ Your analysis should produce: ## 1. Recurring Finding Patterns -Identify clusters of similar findings that appear across multiple reviews. For each pattern: +Identify clusters of similar findings that appear across multiple reviews. Apply evidence thresholds: +- 1-2 occurrences: note as a data point only, do not recommend action +- 3-5 occurrences: emerging pattern, suggest as a candidate guideline with moderate confidence +- 6+ occurrences: strong signal, recommend specific guideline text with high confidence + +For each pattern: - Name the pattern concisely - Count how many reviews contain it - Give 1-2 representative examples with file/line references if available @@ -48,7 +53,10 @@ Identify patterns the reviews keep flagging that are NOT mentioned in the curren ## 5. Suggested Guideline Additions -Provide concrete text snippets that could be added to the project's review_guidelines configuration. Format each as a ready-to-use guideline entry. +Only suggest guidelines backed by 3+ occurrences from section 1 or 3+ noise candidates from section 3. For each, provide: +- The concrete text snippet ready to add to review_guidelines configuration +- The occurrence count that justifies this addition +- Whether this suppresses noise or codifies a real quality standard Be specific and actionable. Avoid vague recommendations. Base everything on evidence from the provided reviews.` From 55486edd17eead29004cc4e59d59c29cf0a1a044 Mon Sep 17 00:00:00 2001 From: Marius van Niekerk Date: Wed, 25 Mar 2026 15:37:50 -0400 Subject: [PATCH 7/7] =?UTF-8?q?=F0=9F=94=AC=20Gracefully=20degrade=20inten?= =?UTF-8?q?t-alignment=20check=20for=20vague=20commit=20messages?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Short or meaningless commit messages like "fix", "wip", or "update" don't carry enough signal for an intent-implementation comparison. When the message is vague, the reviewer now infers intent from the diff itself and skips the alignment check rather than fixating on a low-information message. Co-Authored-By: Claude Opus 4.6 (1M context) --- internal/prompt/prompt.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/internal/prompt/prompt.go b/internal/prompt/prompt.go index 24f0147a..8648b948 100644 --- a/internal/prompt/prompt.go +++ b/internal/prompt/prompt.go @@ -30,11 +30,11 @@ If you use tools while reviewing, finish all tool use before emitting the final // SystemPromptSingle is the base instruction for single commit reviews const SystemPromptSingle = `You are a code reviewer. Review the git commit shown below. -First, read the commit message to understand the developer's intent. Then read the diff and check whether the code changes fully and correctly achieve that intent. Gaps between stated intent and actual implementation are high-value findings. +First, read the commit message to understand the developer's intent. If the commit message is descriptive, check whether the diff fully and correctly achieves that intent — gaps between stated intent and actual implementation are high-value findings. If the commit message is short or vague (e.g. "fix", "wip", "update"), infer intent from the diff itself and skip the intent-alignment check. Check for: -1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit message claims? +1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit message claims? (Skip if the commit message is too vague to make a meaningful comparison.) 2. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions 3. **Security**: Injection vulnerabilities, auth issues, data exposure 4. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps @@ -97,11 +97,11 @@ If you find no issues, state "No issues found." after the summary.` // SystemPromptRange is the base instruction for commit range reviews const SystemPromptRange = `You are a code reviewer. Review the git commit range shown below. -First, read the commit messages to understand the developers' intent. Then read the diff and check whether the code changes fully and correctly achieve that intent. Gaps between stated intent and actual implementation are high-value findings. +First, read the commit messages to understand the developers' intent. If the messages are descriptive, check whether the diff fully and correctly achieves that intent — gaps between stated intent and actual implementation are high-value findings. If the messages are short or vague (e.g. "fix", "wip", "update"), infer intent from the diff itself and skip the intent-alignment check. Check for: -1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit messages claim? +1. **Intent-implementation gaps**: Does the diff actually accomplish what the commit messages claim? (Skip if the messages are too vague to make a meaningful comparison.) 2. **Bugs**: Logic errors, off-by-one errors, null/undefined issues, race conditions 3. **Security**: Injection vulnerabilities, auth issues, data exposure 4. **Testing gaps**: Missing unit tests, edge cases not covered, e2e/integration test gaps