Skip to content

feat: add gate-effectiveness evaluation for v1.7.0 - #125

Merged
nugehs merged 2 commits into
mainfrom
segzy/gate-effectiveness-eval
Aug 17, 2026
Merged

feat: add gate-effectiveness evaluation for v1.7.0#125
nugehs merged 2 commits into
mainfrom
segzy/gate-effectiveness-eval

Conversation

@nugehs

@nugehs nugehs commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add otito eval --gate-effectiveness using the real staged local gate in isolated temporary Git repositories
  • prove one valid control passes and six adversarial change-sets fail for named deterministic reasons
  • include the gate corpus in release quality checks and installed-tarball smoke
  • align the evaluation docs with the merged trust-harness positioning from PR docs: lead with independent trust harness #124
  • prepare the v1.7.0 minor release across package.json, package-lock.json, the changelog, and the MCP Registry server.json manifest

Safety boundary

  • only reviewed committed fixtures are copied
  • corpus entries cannot supply commands or redirect execution outside evals/fixtures
  • path-like change-set names are rejected
  • customer repositories are never executed or mutated

Validation

  • npm run ci
  • npm run version:check: npm, lockfile, and MCP manifest agree on 1.7.0
  • npm run eval:gate: 7/7 cases passed; 6/6 expected blocks matched their encoded reasons
  • tests: 398/398 passed
  • coverage: 91.91% lines, 76.22% branches, 93.70% functions
  • production dependency audit: 0 vulnerabilities
  • clean tarball smoke: installed CLI reported v1.7.0; packaged accuracy and gate evals passed

Release boundary

This PR prepares v1.7.0. It does not publish npm, create the Git tag or GitHub Release, or publish the MCP Registry entry; those remain post-merge release actions.

@nugehs nugehs changed the title feat: add gate-effectiveness evaluation feat: add gate-effectiveness evaluation for v1.7.0 Aug 17, 2026
@nugehs
nugehs merged commit 9962cec into main Aug 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant