fix(deploy-preview): scorecard gate misjudges not_applicable, no_raw_secrets, and npm_signatures_verified#60
Merged
jorisjonkers-dev-agents[bot] merged 2 commits intoJul 10, 2026
Conversation
…ets (#59) - no_raw_secrets now reports the offending file path on failure (fail:raw-secret-in:<path>) and correctly passes when out/manifests/ is empty or missing (all-renders-fail case) - npm_signatures_verified is not_applicable in preview mode; --provenance-verified false is always passed so the check can never pass and should not gate on it - Remove the render-failure block that incorrectly overrode no_raw_secrets to fail - Fix render_preview_summary icon to match fail* prefix (not exact "fail" string) - Add tests/test_scorecard.py covering all four bug fixes
The overall gate used jq select(. == "fail") (exact match). With
no_raw_secrets now returning "fail:raw-secret-in:<path>", a real raw
secret would not count as a failure and the gate would wrongly pass.
Switch to startswith("fail") and mirror the filter in the tests via a
shared constant asserted to appear verbatim in run.sh.
jorisjonkers-dev-agents
Bot
added
the
type: bug
jorisjonkers-dev-agents
Bot
changed the title
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #59
Problem
The deploy-validate scorecard gate produced a red scorecard on every fresh service adoption (observed identically on the auth-api, agents-api, and agent-runtime onboardings). Recorded run logs show all five fragment renders failing, after which the gate exited 1 with
no_raw_secrets=failandnpm_signatures_verified=fail— neither of which reflected a genuine finding.Root causes
no_raw_secretsoverridden tofailon any render failure. A block inmain()forcedno_raw_secrets=failwhenever any fragment render failed. An emptyout/manifests/cannot contain secrets; the underlying grep check already passes correctly in that case. The override is removed — render failures still emit per-fragment::warningdiagnostics.npm_signatures_verifiedstructurally impossible to pass in preview. The action always invokesartifact emit-contractwith--provenance-verified false(no published artifact exists at PR time), so the contract check failed on every run. The check is nownot_applicablein preview mode; provenance verification belongs to the deploy-artifact gate.no_raw_secretsnow reportsfail:raw-secret-in:<path>naming the first offending manifest file. The overall gate filter switched from exact== "fail"matching tostartswith("fail")so detail-suffixed failures still fail the gate, and the preview-summary icon logic matches thefail*prefix.not_applicablewas already excluded from the overall fail computation; the three SC-11 checks that define it (route_owner_authmode_declared,stateful_policy_declared,raw_manifests_guarded) now have explicit test coverage provingnot_applicablecannot fail the gate.Tests
New
tests/test_scorecard.py(18 tests) sourcesrun.shand exercisescompute_scorecarddirectly:not_applicablevalues never count as failures; a minimal fresh-adoption fixture yields a green scorecard end to endnpm_signatures_verifiedisnot_applicableregardless of contract contentno_raw_secretspasses whenout/manifests/is absent, empty, or secret-free — including when render failures are simulatedno_raw_secretsfailure names the offending file pathfail:values fail the overall gate; the jq gate filter asserted in tests must appear verbatim inrun.shso the two cannot drift