Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions plugins/govkit/skills/govkit-feature-refine/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ fields:
acceptance_criteria: optional
nfrs: optional
evaluation_criteria: optional
agentic_behavior: optional (yes | no, only if the team has stated it explicitly)
assumptions: optional
open_questions: optional
out_of_scope: optional
Expand Down Expand Up @@ -175,9 +176,9 @@ Open with a short prose summary of the feature — one or two paragraphs, no bul

Do not add unsupported details. If key inputs are missing (no acceptance criteria, no description), say so plainly in the summary.

Then STOP. End the message by confirming the summary matches the team's understanding and asking whether to continue to the full analysis, for example: "Does this match your understanding of the feature? Say 'continue' and I'll run the full review." Do not reveal blockers, scores, or the Development Token recommendation at this stage — the summary checkpoint exists so the team corrects misread intent *before* the analysis is anchored on it. If the user corrects the summary, restate the corrected understanding and offer to continue again.
Then STOP. End the message with two things: confirmation that the summary matches the team's understanding, and one explicit closed question — **"Agentic behavior: yes or no?"** — meaning: will this feature use AI agents that act autonomously (planning, multi-step tool use, orchestration of multiple agents, or agent-to-agent handoffs), as opposed to no AI at all or a single-shot AI call? For example: "Does this match your understanding of the feature? And one explicit check: agentic behavior — yes or no? Say 'continue' and I'll run the full review." The team's answer sets `multi_agent` in the Evaluation Criteria output during Step 8; ask the question even when the answer looks obvious from the spec — the team's explicit answer is the record, not your inference. Do not reveal blockers, scores, or the Development Token recommendation at this stage — the summary checkpoint exists so the team corrects misread intent *before* the analysis is anchored on it. If the user corrects the summary, restate the corrected understanding and offer to continue again.

Skip the pause only when the user explicitly asks for the full review in one pass (for example "go straight to the decision" or "no need to pause"). Even then, still open the response with the prose summary before the analysis.
Skip the pause only when the user explicitly asks for the full review in one pass (for example "go straight to the decision" or "no need to pause"). Even then, still open the response with the prose summary before the analysis, and if the input does not answer the agentic behavior question, carry it into the full analysis as a refinement question (see Step 8).

### Stage 2 — Full analysis (on user confirmation)

Expand Down Expand Up @@ -311,6 +312,18 @@ For GenAI behavior, inspect:

Do not invent thresholds. Ask for them or mark gaps.

#### Agentic behavior question and the `multi_agent` flag

Every review asks the explicit question **"Agentic behavior: yes or no?"** — normally at the Stage 1 checkpoint (Step 2). The team's answer sets the `multi_agent` field in the feature's `eval_criteria.yaml`:

| Team's answer | `eval_criteria.yaml` |
|---|---|
| Yes | `multi_agent: true` |
| No | `multi_agent: false` |
| Not answered yet | Leave `multi_agent` unset and list the question under Questions for refinement (QA / evaluation) |

Set the flag only from the team's explicit yes/no response — never infer it from the spec, even when the answer seems obvious. When the answer is yes, also confirm the evaluation criteria cover tool or agent routing and the human review path.

### Step 9: Score the draft (advisory)

Score only after understanding the feature, reviewing scenarios, and identifying blockers. The score is a diagnostic that shows where the draft is weak. It does not by itself authorize coding — blockers do that.
Expand Down Expand Up @@ -421,6 +434,7 @@ Approved | Approved with edits | Blocked
```

### Evaluation Criteria
<!-- Must include `multi_agent: true|false`, set from the team's explicit answer to the agentic behavior question (Step 8) — never inferred. If the question is unanswered, omit this section and keep it in Questions for refinement. -->
```yaml
<copy-ready eval criteria>
```
Comment on lines 436 to 440

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Eval criteria omission conflict 🐞 Bug ≡ Correctness

SKILL.md instructs omitting the entire “Evaluation Criteria” copy-ready block when agentic behavior
is unanswered, but Step 8 says to leave only multi_agent unset and keep the question under
refinement. This inconsistency can result in teams not producing/updating eval_criteria.yaml,
which downstream metrics code treats as missing/invalid and will reduce completeness scoring.
Agent Prompt
## Issue description
The SKILL output template currently says to omit the entire **Evaluation Criteria** section if the agentic-behavior question is unanswered, which conflicts with Step 8 guidance that says to leave only `multi_agent` unset. This can lead teams to not produce a copy-ready `eval_criteria.yaml` update at all.

## Issue Context
Downstream tooling parses `features/<id>/eval_criteria.yaml` and treats it as missing/invalid when it is absent or not parseable; omitting the copy-ready block increases the chance that the file is never created/updated.

## Fix Focus Areas
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[315-325]
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[436-440]

## Suggested change
Update the template comment to:
- Keep the **Evaluation Criteria** section present.
- When unanswered, emit copy-ready YAML **without** a `multi_agent` key (or explicitly state “omit the `multi_agent` line”), and list the unanswered question under “Questions for refinement (QA / evaluation)”.
- Ensure wording matches the Step 8 table (“leave `multi_agent` unset”, not “omit this section”).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Expand All @@ -436,7 +450,7 @@ If the input is incomplete, shorten the output and focus on questions, blockers,

## Question rules

Ask no more than 5 high-priority questions unless the user requests a full backlog.
Ask no more than 5 high-priority questions unless the user requests a full backlog. The agentic behavior yes/no question (Step 8) is structural and always asked; it does not count against this budget.

Comment on lines 451 to 454

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

2. Wrong step number reference 🐞 Bug ⚙ Maintainability

The question budget rule calls the agentic-behavior question “Step 8”, but the question is asked at
Stage 1 Step 2 (Step 8 only maps the answer to multi_agent). This internal mismatch can confuse
maintenance and prompt-following behavior.
Agent Prompt
## Issue description
In the Question rules section, the agentic behavior question is referenced as “Step 8”, but it is asked at Stage 1 Step 2.

## Issue Context
Step 8 is where the mapping guidance lives; the actual question is asked at the Stage 1 checkpoint.

## Fix Focus Areas
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[167-182]
- plugins/govkit/skills/govkit-feature-refine/SKILL.md[451-454]

## Suggested change
Change “(Step 8)” to “(Stage 1, Step 2)” or remove the step number entirely (e.g., “The agentic behavior yes/no question is structural…”).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Prioritize questions in this order:

Expand Down Expand Up @@ -497,6 +511,7 @@ Do not:
- Invent NFR thresholds
- Add implementation design into Gherkin
- Create sample scenarios during real-work pilots
- Set `multi_agent` in eval_criteria.yaml without the team's explicit yes/no answer
- Replace PM, QA, or Engineering judgment
- Start coding tasks
- Create step definitions
Expand All @@ -505,6 +520,7 @@ Do not:
Always:

- Preserve product intent
- Ask the agentic behavior question explicitly and record the answer as `multi_agent`
- Surface uncertainty
- Separate blockers from improvements
- Keep refinement questions actionable
Expand Down
12 changes: 10 additions & 2 deletions plugins/govkit/skills/govkit-feature-refine/evals/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
"files": [
"files/feature-ai-summary-missing-evals.md"
],
"expected_output": "Decision must be Blocked because evaluation criteria are missing for AI behavior (a critical blocker per the rubric). The review should call for accuracy/groundedness/safety evaluation criteria with thresholds, data sources, and evidence artifacts \u2014 but must ask for thresholds or mark them as gaps, never invent numeric thresholds. The Gherkin itself is reasonable (including the failure-path rule), so the review should not manufacture Gherkin problems to justify the block.",
"expected_output": "Decision must be Blocked because evaluation criteria are missing for AI behavior (a critical blocker per the rubric). The review should call for accuracy/groundedness/safety evaluation criteria with thresholds, data sources, and evidence artifacts \u2014 but must ask for thresholds or mark them as gaps, never invent numeric thresholds. The Gherkin itself is reasonable (including the failure-path rule), so the review should not manufacture Gherkin problems to justify the block. Because the pause was skipped and the spec does not answer it, the review should surface the explicit 'agentic behavior: yes/no' question as a refinement question and must not set multi_agent: true or false in eval_criteria.yaml without the team's answer.",
"assertions": [
{
"name": "decision-is-blocked",
Expand All @@ -83,6 +83,10 @@
{
"name": "genai-dimensions-named",
"text": "The evaluation gaps reference GenAI dimensions such as accuracy, groundedness, safety, or human review path."
},
{
"name": "agentic-question-not-inferred",
"text": "The review surfaces the explicit 'agentic behavior: yes/no' question for the team and does not emit multi_agent: true or multi_agent: false in eval_criteria.yaml without the team's answer."
}
]
},
Expand All @@ -93,7 +97,7 @@
"files": [
"files/feature-invoice-approval-strong.md"
],
"expected_output": "Stage 1 only: a short prose summary (one or two paragraphs, no bullets, no headings) covering the actor, business outcome, key rules, boundaries, and evidence expectations \u2014 then a stop that asks the user to confirm the summary and say 'continue' for the full analysis. The response must NOT contain a readiness score, a Development Token decision, a blocker list, or the full review template. The full analysis comes only after the user confirms.",
"expected_output": "Stage 1 only: a short prose summary (one or two paragraphs, no bullets, no headings) covering the actor, business outcome, key rules, boundaries, and evidence expectations \u2014 then a stop that asks the user to confirm the summary and say 'continue' for the full analysis. The checkpoint must also ask the explicit 'agentic behavior: yes or no?' question, whose answer later sets multi_agent in eval_criteria.yaml. The response must NOT contain a readiness score, a Development Token decision, a blocker list, or the full review template. The full analysis comes only after the user confirms.",
"assertions": [
{
"name": "summary-is-prose",
Expand All @@ -110,6 +114,10 @@
{
"name": "summary-is-accurate",
"text": "The prose summary accurately reflects the invoice approval feature (threshold routing, rejection with reason, role restriction, audit evidence) without inventing details."
},
{
"name": "asks-agentic-question",
"text": "The checkpoint explicitly asks the agentic behavior yes/no question (whether the feature will use autonomous/agentic AI behavior)."
}
]
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,7 @@ Ready signals:
- Pass/fail threshold is stated.
- Evidence artifact is named.
- Regression expectations are stated.
- The agentic behavior question was asked explicitly, and the team's yes/no answer is recorded as `multi_agent: true|false` in `eval_criteria.yaml`.

Evaluation areas:

Expand All @@ -347,6 +348,7 @@ Evaluation areas:

Questions:

- Agentic behavior: yes or no? (The explicit answer sets `multi_agent: true|false` in `eval_criteria.yaml` — never infer it.)
- What does good output mean for this feature?
- What dataset or examples prove it?
- What threshold decides pass or fail?
Expand Down
Loading