Skip to content

ci: stop security-scan crashing on fork PRs; de-flake trust payload test#298

Merged
prakashUXtech merged 1 commit into
devfrom
ci/harden-security-scan-and-flaky-test
Jul 10, 2026
Merged

ci: stop security-scan crashing on fork PRs; de-flake trust payload test#298
prakashUXtech merged 1 commit into
devfrom
ci/harden-security-scan-and-flaky-test

Conversation

@prakashUXtech

Copy link
Copy Markdown
Contributor

What

Two CI failures were landing on contributor PRs, and neither was caused by the contributor's code. Both are fixed here.

1. security-scan fails on PRs from forks (#295, #296)

The dangerous-pattern scan grep used eval\s*\(, which matches retrieval( (and set_retrieval_weight(, ...) all over the memory modules. So an innocent diff touching manager.py or soul.py set FOUND=1, and the step then called gh pr comment. Fork PRs run with a read-only GITHUB_TOKEN, so the comment returns GraphQL: Resource not accessible by integration (addComment), and under set -e that failed the whole security-scan job. It surfaced as a red security check on PRs that had no security issue.

Fixes:

  • Word-boundary the eval / exec patterns (\beval\s*\(), so they match real eval( / obj.eval( calls but not retrieval(. Verified against a fixture.
  • Guard both gh pr comment calls with a || echo "::warning::..." fallback, so a fork-token 403 degrades to a log warning instead of crashing the job. This is the same fork-token fix ci: run tests on dev PRs and stop the quality gate crashing on forks #280 applied to the check-quality job, now extended to security-scan. Real-secret detection still exits non-zero.

2. Flaky trust test (#297)

test_payload_is_hashed_not_stored asserted assert "t1" not in raw, where raw is a serialized chain that includes the base64 Ed25519 signature. The signature covers a wall-clock timestamp, so it changes every run, and roughly 2% of the time the base64 happens to contain the 2-char substring t1. That is what failed #297's run (the signature was ...xfatt1Jo...); the PR just rolled the unlucky 2%. The same suite passed on dev and on #295/#296 by luck.

Fix: use long hyphenated sentinels (do-not-leak-token-1) that cannot appear in base64 (no hyphen in the alphabet) or hex output. Ran the test 40x locally with zero failures; by construction it can no longer false-match.

Why now

Sidhant's #295/#296/#297 (which fix #281/#282/#283) were blocked by red CI that had nothing to do with their changes. This unblocks them and every future fork PR.

Testing

  • YAML parses; all three security-scan shell blocks pass bash -n.
  • \beval\s*\( matches eval(code) and obj.eval(s), not retrieval(q) / set_retrieval_weight(.
  • The de-flaked test passes 40/40 locally.

Two CI failures were hitting contributor PRs, neither caused by their code:

security-scan (fails on PRs from forks): the dangerous-pattern grep used
'eval\s*\(', which matches 'retrieval(' all over the memory modules, so it
set FOUND=1 on innocent diffs and then called 'gh pr comment'. Fork PRs get a
read-only GITHUB_TOKEN, so the comment 403s, and under 'set -e' that failed the
whole job (reported as a real security failure). Word-boundary the eval/exec
patterns so they only match real calls, and guard both gh pr comment calls so a
403 warns instead of crashing (same fork-token fix #280 applied to the
check-quality job, now extended to security-scan).

test_payload_is_hashed_not_stored: asserted 'assert "t1" not in raw' over a
serialized chain that includes the base64 Ed25519 signature. The signature
covers a wall-clock timestamp, so it varies per run and ~2% of the time the
base64 contains the 2-char substring 't1' -> flaky failure unrelated to the
change under test. Use long hyphenated sentinels that can't occur in base64 or
hex output.
@github-actions

Copy link
Copy Markdown

Heads up

  • This PR touches sensitive files (CI/CD workflows). These paths require maintainer review via CODEOWNERS.

Please update your PR to address these points.

@prakashUXtech
prakashUXtech merged commit 1fc3b68 into dev Jul 10, 2026
6 checks passed
@prakashUXtech
prakashUXtech deleted the ci/harden-security-scan-and-flaky-test branch July 10, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant