Skip to content

test(security): bind the file census to the guard's EMITTED output, not to a return value - #341

Merged
andrei-hasna merged 1 commit into
mainfrom
fix/38d15243-census-output-regression
Aug 2, 2026
Merged

test(security): bind the file census to the guard's EMITTED output, not to a return value#341
andrei-hasna merged 1 commit into
mainfrom
fix/38d15243-census-output-regression

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Remediation on the review of #340 (merged). Test-only — the guard itself is untouched by this PR.

The defect

The census change in #340 shipped undefended. Deleting (${census}) from the fail path's console.error left the suite fully green:

rc_test=0
19 pass
0 fail

expect(scanned).toBe(2) reads scanPaths()'s return value. A returned number and a printed line are different things, and only one of them is the deliverable. I reproduced the reviewer's mutation independently before fixing it and got the same green.

This is the identical hole #340 argued about the arity half — asserting a positive that cannot distinguish the fixed state from the broken one. I made that argument and then failed to apply it to my own change.

Why it warrants a real regression rather than a note

The CI step name Package-manager secret guard reads identically before and after the scan was widened from a basename allow-list to the whole text tree. A green step proves a guard ran, never which guard ran. The count of files opened — roughly 1,220 then, 22,745 now — is the only observable that changed, and a release gate outside this repository keys on scanned > 20000 in this output. If the census silently stops printing on the fail path, that external gate reads the absence as a pass. A number that is load-bearing elsewhere needs a test that fails when it goes missing.

What the tests do

They run the real entrypoint against a synthetic git-tracked package — a git index plus a package.json, so the guard's git ls-files union with npm pack --dry-run is exercised the way a publish would — and assert the emitted text:

  • the census is present in the fail path's stderr, not just in a return value;
  • the number is the true file count (2), so it cannot be satisfied by a hardcoded literal;
  • the report still never echoes what it matched;
  • the pass path carries it too.

Verified in both directions

vs main 134cf246 (fail path had no census)        EXITCODE=1   20 pass / 1 fail
vs the reviewer's exact mutation on the fix       EXITCODE=1   20 pass / 1 fail
restored                                          EXITCODE=0   21 pass / 0 fail

Suite 3817 pass / 190 fail against a 3815 / 190 baseline measured on HEAD~1 — identical failure sets, the +2 are these tests. bun run typecheck exits 0. Sentinels are synthetic; 0 of the naming standard's 18 example names and 0 of the 2 scrubbed connector literals appear in the added lines.

Note on how this PR is based

#340 was merged while this remediation was in flight, and my push re-created the auto-deleted branch. Rather than PR that stale branch — whose diff would have reverted the bun run build && bun test change that landed in e8128814 — I cut a fresh branch from current main and cherry-picked the test commit. The diff against main is 74 added lines in one file.

Refs: todos 38d15243


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ot to a return value

Remediation on the review of #340. The census change shipped undefended: deleting
`(${census})` from the fail path's console.error left the suite fully green at
19 pass / 0 fail. The existing `expect(scanned).toBe(2)` assertions read
scanPaths()'s RETURN VALUE, and a returned number and a printed line are
different things — only one of them is the deliverable.

This is the same hole this branch already argued about the arity half: asserting
a positive that cannot distinguish the fixed state from the broken one. I made
that argument and then did not apply it to my own change.

It is worth a real regression rather than a note, because the number is
load-bearing OUTSIDE this repository. The CI step name reads identically before
and after the scan was widened from a basename allow-list to the whole text tree,
so a green step proves a guard ran and never which guard ran; the count of files
opened is the only observable that changed, and a release gate elsewhere keys on
`scanned > 20000` in this output. If the census silently stops printing on the
fail path, that gate reads the absence as a pass.

The new tests run the REAL entrypoint against a synthetic git-tracked package —
git index plus package.json, so the guard's `git ls-files` union with
`npm pack --dry-run` is exercised as a publish would — and assert the emitted
text. They check the count is present, that it is the true file count rather than
a hardcodable literal, and that the report still never echoes what it matched.

Verified in both directions rather than asserted. Against main 134cf24, whose
fail path carries no census: 20 pass / 1 fail. Against the reviewer's exact
mutation on the fixed file: 20 pass / 1 fail. Restored: 21 pass / 0 fail. Suite
3817 pass / 190 fail against a 3813 / 190 baseline on main — identical failure
sets, the +4 are this branch's tests. Typecheck clean.

Test-only change; the guard itself is untouched by this commit.

Refs: todos 38d15243

Agent: Silvanus
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

[REVIEW] GO — #341 @ f939144 — lens: correctness+security+gates, reviewer unresolved-account001 (1 of 1)

Reviewed exact candidate:

  • Confirmed HEAD f9391447fb0a2bdae1dd3bb967a07977201f43b4 and fetched base origin/main at 12a9018be4312ddd4d787bf5e5b526e3471ed06a.
  • Ran git log --oneline origin/main..HEAD (exit 0), git diff origin/main...HEAD --stat (exit 0), and read the full diff.
  • Read the complete changed scripts/check-package-secrets.test.ts, the surrounding/complete scripts/check-package-secrets.ts entrypoint and scan path, and the gate declarations in package.json.
  • Traced both added end-to-end fixtures through git ls-files + npm pack --dry-run, scanPaths, and the emitted stdout/stderr census. The sentinels are synthetic and the failure-path assertion verifies the matched value is not emitted.

Commands and results:

  • bun install — exit 0. Setup only, not counted as the repository test gate; 183 root packages and 164 dashboard packages installed.
  • bun run typecheck — exit 0. Pass/fail counts: not emitted by the declared TypeScript gate.
  • bun run test — exit 0. Build completed; 4,007 tests passed, 0 failed, across 454 files (35,259 expectations).
  • git diff --check origin/main...HEAD — exit 0.

Blocking P0/P1 findings: none.

Non-blocking follow-ups: none.

Verdict: GO. The added tests exercise the real guard entrypoint on both success and failure output paths and correctly bind the observable file census without leaking the matched sentinel.

@andrei-hasna
andrei-hasna merged commit 05c44db into main Aug 2, 2026
2 checks passed
@andrei-hasna
andrei-hasna deleted the fix/38d15243-census-output-regression branch August 2, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant