Skill engine: weighted scoring with negative patterns and context guards

[AnExiledDev]

Summary

• Weighted phrase scoring — phrases carry confidence weights (0.0–1.0) instead of binary match/no-match. Specific phrases like "build a fastapi app" score 1.0; ambiguous phrases like "start building" score 0.2
• Negative patterns — substrings that instantly disqualify a skill (e.g., "pydanticai" suppresses fastapi to resolve the pydantic overlap)
• Context guards — low-confidence matches (score < 0.6) require a confirming context word elsewhere in the prompt. "health check" only suggests docker if "docker", "container", or "compose" also appears
• Ranked results capped at 3 — suggestions sorted by score (then priority tier), max 3 returned. Eliminates 6+ skill suggestion floods
• Priority tiers — explicit commands (10) > technology skills (7) > patterns (5) > generic (3) for tie-breaking

False positives fixed

Prompt	Before	After
"start building the UI"	spec-build	✗ blocked (no "spec" context)
"plan a feature roadmap"	spec-new	✗ blocked (no "spec" context)
"find error in my regex"	debugging	✗ blocked (no log/crash/bug context)
"health check endpoint for the API"	docker	✗ blocked (no docker context)
"upgrade to the latest version of node"	migration-patterns	✗ blocked (no framework context)
"update the docs quickly"	documentation-patterns	✗ blocked (no doc-specific context)

True positives preserved

All existing matches for explicit prompts ("build a fastapi app", "write pytest tests", "create a dockerfile", "spec-build", etc.) continue to work.

Test plan

• 28/28 test matrix: 10 false positive cases blocked, 18 true positive cases preserved
• MAX_SKILLS cap verified: 4+ matches → only top 3 returned
• Pydantic overlap: fastapi vs pydantic-ai cleanly resolved via negative patterns
• End-to-end hook pipe: valid JSON output shape unchanged
• Edge cases: empty input, no matches, single match all produce correct behavior

Summary by CodeRabbit

• Changed

  • Skill auto-suggestion now uses weighted scoring with confidence levels instead of binary matching.
  • Suggestions are ranked by score and priority tier, limited to top 3 results.
  • Added negative patterns to prevent false suggestion triggers.
  • Context guards require confirming words for low-confidence matches.

• Documentation

  • Updated skill-engine documentation reflecting the new weighted scoring algorithm.

[coderabbitai]

Warning

Rate limit exceeded

@AnExiledDev has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 7 minutes and 17 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 1d59577 and 991a463.

📒 Files selected for processing (1)

• .devcontainer/plugins/devs-marketplace/plugins/skill-engine/scripts/skill-suggester.py

📝 Walkthrough

Walkthrough

The changes introduce a weighted-scoring skill suggestion system that replaces binary matching with continuous confidence scores (0.0–1.0), adds negative pattern disqualification, enforces context guards for low-confidence matches, ranks results by score and priority tier, and caps suggestions to three skills per prompt.

Changes

Cohort / File(s)	Summary
Documentation .devcontainer/CHANGELOG.md, .devcontainer/plugins/devs-marketplace/plugins/skill-engine/README.md	Added Unreleased section documenting new weighted scoring features, including continuous confidence weights, negative patterns, context guards, ranked results, and priority tiers. Updated README to explain the new auto-suggestion behavior, skill keyword definitions (phrases with weights, regex terms, negative patterns, context guards), and updated hook lifecycle steps.
Core Implementation .devcontainer/plugins/devs-marketplace/plugins/skill-engine/scripts/skill-suggester.py	Replaced naive substring/regex matching with a weighted-scoring pipeline. Introduced module-level constants for scoring thresholds and caps (MAX_SKILLS=3, MIN_SCORE, TERM_WEIGHT, CONTEXT_GUARD_THRESHOLD). Restructured SKILLS data to store phrases as (phrase, weight) tuples with negative patterns and context guards. Implemented centralized _score_skill() function for scoring logic. Rewrote match_skills() to aggregate scores, sort by score then priority, and return up to MAX_SKILLS suggestions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 Whiskers twitching with delight,
Weighted scores now shine so bright!
Context guards and patterns sly,
Ranking skills that multiply—
Three little suggestions, ranked with care,
A scoring magic in the air! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and accurately summarizes the main change: implementing weighted scoring with negative patterns and context guards in the skill engine, which is the core focus across all modified files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/skill-suggester-scoring

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In
@.devcontainer/plugins/devs-marketplace/plugins/skill-engine/scripts/skill-suggester.py:
- Around line 190-191: The negative pattern "DockerClient" in the negative list
is mixed-case but the matching logic uses a lowercased input (see the check `if
neg in lowered`), so change the pattern to lowercase ("dockerclient") to ensure
it can match; update the entry in the negative array where "DockerClient"
appears so all entries are lowercase and consistent with the matching performed
by the code that checks `neg in lowered`.

---

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d2ba55e and 1d59577.

📒 Files selected for processing (3)

• .devcontainer/CHANGELOG.md
• .devcontainer/plugins/devs-marketplace/plugins/skill-engine/README.md
• .devcontainer/plugins/devs-marketplace/plugins/skill-engine/scripts/skill-suggester.py