This product only earns trust if its own validation behavior is stable, reproducible, and well-tested. Testing is therefore part of the product, not just the engineering process behind it.
- manifest parsing
- path normalization
- rule evaluation logic
- severity and score calculations
- valid plugin package
- missing manifest fields
- broken skill references
- malformed MCP config
- context-bloated tool schemas
- successful stdio startup
- startup timeout
- server crash on boot
- invalid
tools/listresponse
Maintain a curated set of package fixtures under source control with:
- expected findings
- expected score bands
- expected exit codes
This protects report stability as rules evolve.
Every change should run:
- lint
- typecheck
- unit tests
- fixture integration tests
- selected runtime probe tests
- Merge into the release branch.
- Run the full validation suite.
- Build distributable CLI artifacts.
- Publish package version.
- Publish release notes with rule changes and breaking behavior.
- Run the release sync gate for the lane:
npm run verify-release-syncfor stable releases, ornpm run verify-release-sync -- --dist-tag next --prereleasefor release candidates.
Use semantic versioning.
- patch: bug fixes and non-breaking rule improvements
- minor: new rules and new output fields
- major: output contract changes or rule semantics that may break CI assumptions
- every rule has a documented identifier
- every CLI flag has at least one example
- every release notes document highlights new blocking rules