ci: stop security-scan crashing on fork PRs; de-flake trust payload test#298
Merged
Merged
Conversation
Two CI failures were hitting contributor PRs, neither caused by their code:
security-scan (fails on PRs from forks): the dangerous-pattern grep used
'eval\s*\(', which matches 'retrieval(' all over the memory modules, so it
set FOUND=1 on innocent diffs and then called 'gh pr comment'. Fork PRs get a
read-only GITHUB_TOKEN, so the comment 403s, and under 'set -e' that failed the
whole job (reported as a real security failure). Word-boundary the eval/exec
patterns so they only match real calls, and guard both gh pr comment calls so a
403 warns instead of crashing (same fork-token fix #280 applied to the
check-quality job, now extended to security-scan).
test_payload_is_hashed_not_stored: asserted 'assert "t1" not in raw' over a
serialized chain that includes the base64 Ed25519 signature. The signature
covers a wall-clock timestamp, so it varies per run and ~2% of the time the
base64 contains the 2-char substring 't1' -> flaky failure unrelated to the
change under test. Use long hyphenated sentinels that can't occur in base64 or
hex output.
Heads up
Please update your PR to address these points. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two CI failures were landing on contributor PRs, and neither was caused by the contributor's code. Both are fixed here.
1.
security-scanfails on PRs from forks (#295, #296)The dangerous-pattern scan grep used
eval\s*\(, which matchesretrieval((andset_retrieval_weight(, ...) all over the memory modules. So an innocent diff touchingmanager.pyorsoul.pysetFOUND=1, and the step then calledgh pr comment. Fork PRs run with a read-onlyGITHUB_TOKEN, so the comment returnsGraphQL: Resource not accessible by integration (addComment), and underset -ethat failed the wholesecurity-scanjob. It surfaced as a red security check on PRs that had no security issue.Fixes:
eval/execpatterns (\beval\s*\(), so they match realeval(/obj.eval(calls but notretrieval(. Verified against a fixture.gh pr commentcalls with a|| echo "::warning::..."fallback, so a fork-token 403 degrades to a log warning instead of crashing the job. This is the same fork-token fix ci: run tests on dev PRs and stop the quality gate crashing on forks #280 applied to thecheck-qualityjob, now extended tosecurity-scan. Real-secret detection still exits non-zero.2. Flaky trust test (#297)
test_payload_is_hashed_not_storedassertedassert "t1" not in raw, whererawis a serialized chain that includes the base64 Ed25519 signature. The signature covers a wall-clock timestamp, so it changes every run, and roughly 2% of the time the base64 happens to contain the 2-char substringt1. That is what failed #297's run (the signature was...xfatt1Jo...); the PR just rolled the unlucky 2%. The same suite passed on dev and on #295/#296 by luck.Fix: use long hyphenated sentinels (
do-not-leak-token-1) that cannot appear in base64 (no hyphen in the alphabet) or hex output. Ran the test 40x locally with zero failures; by construction it can no longer false-match.Why now
Sidhant's #295/#296/#297 (which fix #281/#282/#283) were blocked by red CI that had nothing to do with their changes. This unblocks them and every future fork PR.
Testing
security-scanshell blocks passbash -n.\beval\s*\(matcheseval(code)andobj.eval(s), notretrieval(q)/set_retrieval_weight(.