Add explicit agentic-behavior question to feature-refine review#4
Merged
Conversation
The Stage 1 checkpoint now asks "Agentic behavior: yes or no?" and the team's explicit answer sets multi_agent: true|false in the feature's eval_criteria.yaml (Step 8). Unanswered stays unset and moves to Questions for refinement; the flag is never inferred from the spec. QA evidence checklist and skill evals updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR Summary by QodoAdd explicit agentic-behavior question and persist multi_agent in feature refine
AI Description
Diagram
High-Level Assessment
Files changed (3)
|
Code Review by Qodo
1. Eval criteria omission conflict
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
436
to
440
| ### Evaluation Criteria | ||
| <!-- Must include `multi_agent: true|false`, set from the team's explicit answer to the agentic behavior question (Step 8) — never inferred. If the question is unanswered, omit this section and keep it in Questions for refinement. --> | ||
| ```yaml | ||
| <copy-ready eval criteria> | ||
| ``` |
There was a problem hiding this comment.
1. Eval criteria omission conflict 🐞 Bug ≡ Correctness
SKILL.md instructs omitting the entire “Evaluation Criteria” copy-ready block when agentic behavior is unanswered, but Step 8 says to leave only multi_agent unset and keep the question under refinement. This inconsistency can result in teams not producing/updating eval_criteria.yaml, which downstream metrics code treats as missing/invalid and will reduce completeness scoring.
Agent Prompt
## Issue description
The SKILL output template currently says to omit the entire **Evaluation Criteria** section if the agentic-behavior question is unanswered, which conflicts with Step 8 guidance that says to leave only `multi_agent` unset. This can lead teams to not produce a copy-ready `eval_criteria.yaml` update at all.
## Issue Context
Downstream tooling parses `features/<id>/eval_criteria.yaml` and treats it as missing/invalid when it is absent or not parseable; omitting the copy-ready block increases the chance that the file is never created/updated.
## Fix Focus Areas
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[315-325]
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[436-440]
## Suggested change
Update the template comment to:
- Keep the **Evaluation Criteria** section present.
- When unanswered, emit copy-ready YAML **without** a `multi_agent` key (or explicitly state “omit the `multi_agent` line”), and list the unanswered question under “Questions for refinement (QA / evaluation)”.
- Ensure wording matches the Step 8 table (“leave `multi_agent` unset”, not “omit this section”).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
451
to
454
| ## Question rules | ||
|
|
||
| Ask no more than 5 high-priority questions unless the user requests a full backlog. | ||
| Ask no more than 5 high-priority questions unless the user requests a full backlog. The agentic behavior yes/no question (Step 8) is structural and always asked; it does not count against this budget. | ||
|
|
There was a problem hiding this comment.
2. Wrong step number reference 🐞 Bug ⚙ Maintainability
The question budget rule calls the agentic-behavior question “Step 8”, but the question is asked at Stage 1 Step 2 (Step 8 only maps the answer to multi_agent). This internal mismatch can confuse maintenance and prompt-following behavior.
Agent Prompt
## Issue description
In the Question rules section, the agentic behavior question is referenced as “Step 8”, but it is asked at Stage 1 Step 2.
## Issue Context
Step 8 is where the mapping guidance lives; the actual question is asked at the Stage 1 checkpoint.
## Fix Focus Areas
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[167-182]
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[451-454]
## Suggested change
Change “(Step 8)” to “(Stage 1, Step 2)” or remove the step number entirely (e.g., “The agentic behavior yes/no question is structural…”).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The govkit-feature-refine review now asks an explicit "Agentic behavior: yes or no?" question and records the team's answer as
multi_agent: true|falsein the feature'seval_criteria.yaml.Why
Whether a feature involves agentic AI behavior determines which evaluation criteria (agent/tool routing, human review path) apply, and downstream GovKit tooling needs a recorded
multi_agentflag. Inferring it from the spec is unreliable — the team's explicit answer during Draft 0 / 3 Amigos refinement is the record.Changes
SKILL.md: Stage 1 checkpoint (Step 2) now ends with the explicit agentic-behavior yes/no question alongside the summary confirmation; when the pause is skipped, the question carries into the full analysis.SKILL.md: New Step 8 subsection maps the answer — yes →multi_agent: true, no →multi_agent: false, unanswered → leave unset and list under Questions for refinement. A "yes" also triggers a check that agent/tool routing and human review path are covered.SKILL.md: Output template requiresmulti_agentin the copy-ready Evaluation Criteria block (never inferred); normalized input gains an optionalagentic_behaviorfield; question budget and guardrails updated to match.references/qa-evidence-checklist.md: Evaluation criteria review gains a ready signal and leading question for the agentic-behavior declaration.evals/evals.json: eval 4 asserts the question is asked at the checkpoint; eval 3 asserts the question is surfaced when the pause is skipped and thatmulti_agentis not emitted without an answer. An unanswered question is deliberately not a critical blocker, so existing eval decisions are unchanged.Testing
claude plugin validate .passes (same check CI runs).evals.jsonparses as valid JSON.🤖 Generated with Claude Code