Skip to content

fix(quality): avoid substring false positives - #45

Open
chrisknvidia wants to merge 8 commits into
mainfrom
fix/christopherk/issue-30-keyword-boundaries
Open

fix(quality): avoid substring false positives#45
chrisknvidia wants to merge 8 commits into
mainfrom
fix/christopherk/issue-30-keyword-boundaries

Conversation

@chrisknvidia

@chrisknvidia chrisknvidia commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace bare substring checks with boundary-aware matching for quality keywords, reserved names, API documentation, action verbs, and error handling
  • distinguish real XML tags, active MCP usage, actionable README references, actual year references, exclusivity claims, and local nested Markdown links from incidental text
  • keep schema and quality validation consistent and add the complete regression matrix for positive, negative, malformed, hidden, and contextual cases

Documentation parity

  • Updated the Fern Tier 1 quality guide with the new boundary-aware and context-aware semantics.
  • Documented which constructs intentionally affect scoring and which incidental text, examples, comments, external URLs, or unrelated troubleshooting text do not.
  • Added author guidance to fix the reported evidence instead of keyword-stuffing a skill.

Before / after

Input pattern Before After
Reserved word inside a larger token Could change the score by substring Matches only the complete reserved-name segment
XML-looking prose Could be mistaken for markup Only actual XML/HTML-like tags trigger the check
Negated MCP reference Could activate MCP requirements Active, non-negated MCP usage is required
README/external/nested links Link context could be conflated Actionable README and local nested Markdown links are distinguished
Incidental years/action verbs/exclusivity words Could satisfy heuristics inside unrelated text Requires the relevant temporal or instructional context

Validation

  • 142 passed for quality and schema validators on Python 3.12
  • 142 passed for quality and schema validators on Python 3.13
  • 716 passed across all validator tests
  • 3142 passed, 7 skipped, 4 deselected across the repository; the one additional deselected bridge test cannot bind its fixed test port because a pre-existing local process owns 127.0.0.1:18080
  • Ruff lint and formatting checks pass for every changed Python file
  • wheel and source distribution build, Twine strict validation, OSS-boundary scan, staged-diff secret scan, and minimal-wheel vulnerability audit pass
  • installed-wheel CLI smoke tests verify both a false-positive-heavy skill (100/A) and a negative control across CLI, JSON, Markdown, and HTML reports
  • fern check passes with 0 errors

Closes #30

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
Comment thread src/skillevaluator/models/skill.py Outdated
Comment thread src/skillevaluator/validators/quality_score.py
Comment thread src/skillevaluator/validators/quality_score.py
Comment thread src/skillevaluator/validators/quality_score.py

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two boundary/context false positives remain in this revision. Both are reproducible through the quality validator and should be covered before approval.

Comment thread src/skillevaluator/validators/quality_score.py
Comment thread src/skillevaluator/validators/quality_score.py Outdated
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
…e-30-keyword-boundaries

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
rng1995
rng1995 previously approved these changes Aug 12, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the final integrated diff at 3a3405e. Both remaining context false positives are fixed in 30c04be with regression coverage; all review threads are resolved. Local verification passed (158 focused validator tests; 3,175 full-suite tests; Ruff; OSS boundary), and all required GitHub Actions checks are green.

rng1995 and others added 2 commits August 12, 2026 02:36
…e-30-keyword-boundaries

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

# Conflicts:
#	CHANGELOG.md

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after resolving the current main conflict at ec41fbc. The only conflict was CHANGELOG.md; the resolution preserves both PR #45’s boundary/context quality-scoring release note and main’s Tier 3 isolation release note, while all code changes from both sides remain intact. Final audit: git merge-tree is clean, all six review threads are resolved, the branch is mergeable, focused tests pass (158/158), all validator tests pass (732/732), the full suite passes (3753 passed, 13 skipped, 3 deselected), Ruff/diff/changed-file formatting pass, wheel/sdist build and Twine strict/OSS-boundary checks pass, and every final-SHA CI check is green. The transient Security classifier checkout certificate failure passed on the authorized failed-job rerun.

rng1995
rng1995 previously approved these changes Aug 12, 2026
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
@rng1995

rng1995 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

@chrisknvidia - I've already reviewed and approved this PR. Feel free to approve and merge on your end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quality_score.py keyword checks match bare substrings, so some rules fire on correct content and others cannot fail

2 participants