From f2b66a9be35462505d51873d2d06493c7529c1ed Mon Sep 17 00:00:00 2001 From: Rome Thorstenson <36779795+Rome-1@users.noreply.github.com> Date: Fri, 29 May 2026 13:25:04 -0700 Subject: [PATCH 1/4] docs(sable-jnd+dfu): rafter SKILL Setup section + Cursor recipe leads with --local (#147) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real Cursor user report 2026-05-27 showed two install-time friction points the rafter SKILL.md and the Cursor recipe were silent on: 1. A Cursor agent hallucinated `npx rafter-cli agent init --all` (the bare `rafter-cli` resolves to an unrelated npm package — ours is `@rafter-security/cli`). The SKILL.md had no Install / bootstrap section so agents that didn't find rafter on PATH had to guess. 2. The same agent hit a Cursor sandbox prompt for writing `~/.rafter` and had to retry with elevated permissions. `rafter agent init --local --with-cursor` writes `./.rafter/` + `./.cursor/` instead and sidesteps the prompt entirely. Changes: - rafter SKILL.md: replace the bare `## Strengthen the Project` block with a `## Setup` block that names the canonical install commands (`npm i -g @rafter-security/cli`, `pip install rafter-cli`), warns off `npx rafter-cli` (use scoped form), and leads with `rafter agent init --local --with-` for sandboxed sessions. Same content carried into the install-hook / ci init / .rafter.yml / per-platform pointers that the prior section held — single denser paragraph keeps SKILL.md at 118 lines (under the 120-line guard). - node/python parity preserved: `diff -q` on the two SKILL.md files returns clean. - recipes/cursor.md: rebuilt. Now opens with the surface matrix (MCP + hooks + per-skill rules + sub-agent — was just "MCP server" before), then leads with `rafter agent init --local --with-cursor` as the recommended first-run for agent-driven setup, and keeps the global-scope form as the explicit user-scope option. The same install-prerequisite block with the `npx rafter-cli` warning is at the top of the recipe. Tests: 28/28 in `node/tests/brief.test.ts` pass (the `<120 lines` guard still trips on increase, so the SKILL.md edit stays compliant). Closes sable-jnd and sable-dfu. --- node/resources/skills/rafter/SKILL.md | 4 +-- .../resources/skills/rafter/SKILL.md | 4 +-- recipes/cursor.md | 26 ++++++++++++++++--- 3 files changed, 27 insertions(+), 7 deletions(-) diff --git a/node/resources/skills/rafter/SKILL.md b/node/resources/skills/rafter/SKILL.md index aed187f3..89313267 100644 --- a/node/resources/skills/rafter/SKILL.md +++ b/node/resources/skills/rafter/SKILL.md @@ -114,5 +114,5 @@ export RAFTER_API_KEY="..." # or put it in .env Without a key, only `rafter secrets` works — that's secret-hygiene, not code review. If security matters for the task, flag the missing key to the user rather than silently accepting the narrower scan. -## Strengthen the Project -Not wired in yet? `rafter agent install-hook` (pre-commit), `rafter ci init` (CI workflow), `.rafter.yml` (policy). Per-platform setup: `rafter brief setup/`. +## Setup +Not installed? `npm i -g @rafter-security/cli` (Node) or `pip install rafter-cli` (Python). `npx` form is `npx @rafter-security/cli` — the bare `npx rafter-cli` resolves to an unrelated package. Inside Cursor's sandbox or any session where writing `~/.rafter` triggers a prompt, prefer `rafter agent init --local --with-` — writes `./.rafter/` + `././` instead of `$HOME`. Not wired yet? `rafter agent install-hook` (pre-commit), `rafter ci init` (CI), `.rafter.yml` (policy). Per-platform: `rafter brief setup/`. diff --git a/python/rafter_cli/resources/skills/rafter/SKILL.md b/python/rafter_cli/resources/skills/rafter/SKILL.md index aed187f3..89313267 100644 --- a/python/rafter_cli/resources/skills/rafter/SKILL.md +++ b/python/rafter_cli/resources/skills/rafter/SKILL.md @@ -114,5 +114,5 @@ export RAFTER_API_KEY="..." # or put it in .env Without a key, only `rafter secrets` works — that's secret-hygiene, not code review. If security matters for the task, flag the missing key to the user rather than silently accepting the narrower scan. -## Strengthen the Project -Not wired in yet? `rafter agent install-hook` (pre-commit), `rafter ci init` (CI workflow), `.rafter.yml` (policy). Per-platform setup: `rafter brief setup/`. +## Setup +Not installed? `npm i -g @rafter-security/cli` (Node) or `pip install rafter-cli` (Python). `npx` form is `npx @rafter-security/cli` — the bare `npx rafter-cli` resolves to an unrelated package. Inside Cursor's sandbox or any session where writing `~/.rafter` triggers a prompt, prefer `rafter agent init --local --with-` — writes `./.rafter/` + `././` instead of `$HOME`. Not wired yet? `rafter agent install-hook` (pre-commit), `rafter ci init` (CI), `.rafter.yml` (policy). Per-platform: `rafter brief setup/`. diff --git a/recipes/cursor.md b/recipes/cursor.md index 1f9ad41e..53faf291 100644 --- a/recipes/cursor.md +++ b/recipes/cursor.md @@ -1,20 +1,40 @@ # Cursor Setup -Rafter integrates with Cursor through an **MCP server** that exposes scanning and auditing tools. +Rafter's Cursor integration covers an MCP server + Cursor-native hooks (`preToolUse` / `postToolUse` / `beforeShellExecution`) + per-skill rules + a Cursor sub-agent. See [`shared-docs/PLATFORM_PARITY_AUDIT.md`](../shared-docs/PLATFORM_PARITY_AUDIT.md) for the full surface matrix. + +## Install the CLI first + +```sh +npm install -g @rafter-security/cli # Node +# or: +pip install rafter-cli # Python +``` + +> Using `npx`? The canonical form is `npx @rafter-security/cli` — the bare `npx rafter-cli` resolves to an **unrelated** package on npm. ## Automatic setup +### Driven by the Cursor agent itself (recommended for first-run) + +```sh +rafter agent init --local --with-cursor +``` + +Writes to `./.rafter/` and `./.cursor/` instead of `$HOME` — sidesteps Cursor's sandbox prompt for writing under your home directory and scopes the install to this project. Run it from inside the repo you're working in. + +### Global install (one-time, applies to every project) + ```sh rafter agent init --with-cursor ``` -This auto-detects `~/.cursor` and installs the MCP server config. Done. +Auto-detects `~/.cursor` and installs at user scope. Requires elevated permissions if Cursor's sandbox is locked down — the agent will prompt for them. ## Manual setup ### 1. MCP server -Add to your `~/.cursor/mcp.json`: +Add to `~/.cursor/mcp.json` (or `./.cursor/mcp.json` for per-project): ```json { From c993f0d4f28dbb5f6a894d26284f3db1ceab2c88 Mon Sep 17 00:00:00 2001 From: Rome-1 Date: Sun, 31 May 2026 06:34:48 +0000 Subject: [PATCH 2/4] feat(github-action): default severity-threshold to none (report-only) First-install no longer fails the build. Findings still post as PR comments and upload to the Security tab via SARIF, but the workflow exits 0 regardless of severity. When findings exist and the threshold is the default (none), the PR comment includes a one-line tip pointing to severity-threshold: high for opt-in enforcement. Rationale: rafter spans SAST + SCA + secrets where false positives are common; a first install that breaks the build on day one is the worst first impression and gets the action removed. Aligns with the Snyk / CodeQL / Semgrep default of surfacing through Code Scanning + PR comments rather than blocking. Existing users who set severity-threshold explicitly are unaffected. Co-Authored-By: Claude Opus 4.7 --- github-action/README.md | 16 +++++++++++++--- github-action/action.yml | 9 +++++++-- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/github-action/README.md b/github-action/README.md index c683e5b8..e9d94c15 100644 --- a/github-action/README.md +++ b/github-action/README.md @@ -9,7 +9,7 @@ Run [Rafter](https://rafter.so) security scans on your GitHub repositories. Find - Scans on push and pull request events - Posts findings as PR comments with severity breakdown - Uploads SARIF to GitHub Code Scanning (appears in Security tab) -- Configurable severity thresholds for CI gating +- Report-only by default — opt into CI gating with `severity-threshold` - Fast and Plus scan modes ## Quick Start @@ -42,7 +42,7 @@ jobs: |-------|-------------|---------| | `api-key` | Your Rafter API key (required) | — | | `scan-mode` | `fast` or `plus` | `fast` | -| `severity-threshold` | Fail if findings at this level or above: `critical`, `high`, `medium`, `low`, `none` | `high` | +| `severity-threshold` | Fail if findings at this level or above: `critical`, `high`, `medium`, `low`, `none`. Default `none` so first installs don't break builds — set to `high` once the baseline is triaged. | `none` | | `comment-on-pr` | Post results as PR comment | `true` | | `upload-sarif` | Upload SARIF to GitHub Code Scanning | `true` | | `timeout-minutes` | Max wait time for scan completion | `10` | @@ -62,6 +62,17 @@ jobs: ## Examples +### Enforce in CI (recommended after first scan) + +The default install is report-only — findings show up in PR comments and the Security tab but won't fail the check. Once you've triaged the baseline, opt into blocking: + +```yaml +- uses: Raftersecurity/rafter-cli/github-action@main + with: + api-key: ${{ secrets.RAFTER_API_KEY }} + severity-threshold: high # fail on critical/high +``` + ### Block PRs with critical findings only ```yaml @@ -88,7 +99,6 @@ jobs: id: rafter with: api-key: ${{ secrets.RAFTER_API_KEY }} - severity-threshold: none # Don't fail, just report - run: echo "Found ${{ steps.rafter.outputs.findings-count }} issues" ``` diff --git a/github-action/action.yml b/github-action/action.yml index 1929cfc0..32a9b812 100644 --- a/github-action/action.yml +++ b/github-action/action.yml @@ -15,9 +15,9 @@ inputs: required: false default: 'fast' severity-threshold: - description: 'Fail the check if findings at this severity or above are found. Options: critical, high, medium, low, none' + description: 'Fail the check if findings at this severity or above are found. Options: critical, high, medium, low, none. Default "none" so first-install never breaks a build; once you''ve triaged the baseline, set to "high" to enforce.' required: false - default: 'high' + default: 'none' comment-on-pr: description: 'Post scan results as a PR comment' required: false @@ -239,6 +239,7 @@ runs: HIGH_COUNT: ${{ steps.results.outputs.high_count }} MEDIUM_COUNT: ${{ steps.results.outputs.medium_count }} LOW_COUNT: ${{ steps.results.outputs.low_count }} + SEVERITY_THRESHOLD: ${{ inputs.severity-threshold }} run: | MD_REPORT=$(jq -r '.markdown // empty' "${{ runner.temp }}/rafter-results.md" 2>/dev/null || cat "${{ runner.temp }}/rafter-results.md") @@ -278,6 +279,10 @@ runs: echo "" echo "" echo "" + if [ "$FINDINGS_COUNT" -gt 0 ] && [ "$SEVERITY_THRESHOLD" = "none" ]; then + echo "> :information_source: This run is report-only. To fail the build on critical/high findings, set \`severity-threshold: high\` in your workflow." + echo "" + fi echo "---" echo "Scan ID: ${SCAN_ID} | Powered by [Rafter](https://rafter.so)" } >> "$COMMENT_FILE" From daab294b3fbc694fcababcd0dce91f726534c03d Mon Sep 17 00:00:00 2001 From: Rome-1 Date: Sun, 31 May 2026 07:28:33 +0000 Subject: [PATCH 3/4] chore(release): bump to v0.8.3 Headline change in this release is github-action/ cloud variant default-flip from severity-threshold: high to severity-threshold: none (report-only). See CHANGELOG.md for the full entry and the infrastructure improvements that accumulated since 0.8.2. Co-Authored-By: Claude Opus 4.7 --- CHANGELOG.md | 10 ++++++++++ node/package.json | 2 +- node/resources/rafter-security-skill.md | 2 +- python/pyproject.toml | 2 +- python/rafter_cli/resources/rafter-security-skill.md | 2 +- 5 files changed, 14 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 47d7f585..d06cbb6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.8.3] - 2026-05-31 + +### Changed +- **`github-action/` cloud variant: `severity-threshold` default is now `none` (report-only)** (#148). Previously a fresh install with no `severity-threshold` set would fail the build on any critical/high finding — so first-install on a real repo almost always broke CI on day one, the worst possible first impression for a security tool. The default now matches Snyk / CodeQL / Semgrep: scan completes, findings post as a PR comment, SARIF uploads to the Security tab, but the workflow exits 0 regardless of severity. When findings exist under the report-only default, the PR comment includes a one-line tip pointing to `severity-threshold: high` for opt-in enforcement. Workflows that set `severity-threshold` explicitly are unaffected. README updated with an "Enforce in CI (recommended after first scan)" example. Affects only the cloud-API action at `Raftersecurity/rafter-cli/github-action@…`; the root `Raftersecurity/rafter-cli@v…` local-secrets action is unchanged. + +### Infrastructure +- `publish.yaml` migrated to npm Trusted Publishing (OIDC) — no more long-lived `NPM_TOKEN` (#142). Renamed `publish.yml` → `publish.yaml` to match npm's Trusted Publisher manifest, and added `--skip-existing` on the PyPI upload so retries of a partially-failed release no longer error on "File already exists" (#143). +- `node/package.json` now declares `repository` / `homepage` / `bugs` / `license` / `author` metadata so the npm listing renders sidebar links correctly (#145). +- Cursor recipe in the rafter SKILL `Setup` section now leads with `--local` for ephemeral / containerized setups (#147). + ## [0.8.2] - 2026-05-26 ### Changed diff --git a/node/package.json b/node/package.json index bc508446..ea9a38b6 100644 --- a/node/package.json +++ b/node/package.json @@ -1,6 +1,6 @@ { "name": "@rafter-security/cli", - "version": "0.8.2", + "version": "0.8.3", "type": "module", "repository": { "type": "git", diff --git a/node/resources/rafter-security-skill.md b/node/resources/rafter-security-skill.md index 80c609e1..da35e4b0 100644 --- a/node/resources/rafter-security-skill.md +++ b/node/resources/rafter-security-skill.md @@ -1,7 +1,7 @@ --- name: rafter-security description: Security toolkit for AI workflows. Use when scanning code or repos for vulnerabilities, auditing third-party skills/MCPs/agent configs before installing, evaluating shell commands before running them, or generating secure design questions for new features. Provides `rafter run` (remote SAST + SCA, needs RAFTER_API_KEY), `rafter secrets` (offline secrets-only), `rafter agent exec --dry-run` (command-risk classification), and `rafter skill review`. -version: 0.8.2 +version: 0.8.3 homepage: https://rafter.so metadata: openclaw: diff --git a/python/pyproject.toml b/python/pyproject.toml index 57e06b01..52a12b2b 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "rafter-cli" -version = "0.8.2" +version = "0.8.3" description = "Rafter CLI — the default security agent for AI workflows. Free for individuals and open source." authors = ["Rafter Team "] license = "MIT" diff --git a/python/rafter_cli/resources/rafter-security-skill.md b/python/rafter_cli/resources/rafter-security-skill.md index 80c609e1..da35e4b0 100644 --- a/python/rafter_cli/resources/rafter-security-skill.md +++ b/python/rafter_cli/resources/rafter-security-skill.md @@ -1,7 +1,7 @@ --- name: rafter-security description: Security toolkit for AI workflows. Use when scanning code or repos for vulnerabilities, auditing third-party skills/MCPs/agent configs before installing, evaluating shell commands before running them, or generating secure design questions for new features. Provides `rafter run` (remote SAST + SCA, needs RAFTER_API_KEY), `rafter secrets` (offline secrets-only), `rafter agent exec --dry-run` (command-risk classification), and `rafter skill review`. -version: 0.8.2 +version: 0.8.3 homepage: https://rafter.so metadata: openclaw: From c004d173b8681d0cb7fb6d1711e20ecc34c75418 Mon Sep 17 00:00:00 2001 From: Rome-1 Date: Sun, 31 May 2026 19:27:35 +0000 Subject: [PATCH 4/4] test(github-action): add verification suite for v0.8.3 default flip Three pure-bash test scripts exercise the exact code paths touched by the severity-threshold: high -> none default flip: - test-threshold-eval.sh: 30 assertions across all 6 threshold values (critical/high/medium/low/none/unknown-fallback) and finding-count permutations. Confirms 'none' never fails the build regardless of findings, and that critical/high/medium/low still fail as before. - test-pr-comment-tip.sh: 9 assertions on the new report-only-tip block. Confirms the tip appears exactly when (findings > 0) AND (threshold == 'none'). - test-action-yml-defaults.sh: drift detector. Asserts (1) the severity-threshold default is literally 'none', (2) the env var passthrough on the Comment-on-PR step is intact (the GitHub-recommended template-injection mitigation), (3) the tip block is gated on both conditions, (4) the 'none' branch of the threshold-eval doesn't set FAIL=1. Each check independently verified against a mutated copy of action.yml. Wired into a new test-github-action.yml workflow that runs on PRs + main pushes touching github-action/**. The Rafter API is not exercised here; that's a future addition that would need an injectable RAFTER_API_KEY secret and a fixture repo. Co-Authored-By: Claude Opus 4.7 --- .github/workflows/test-github-action.yml | 59 ++++++++++ .../tests/test-action-yml-defaults.sh | 73 ++++++++++++ github-action/tests/test-pr-comment-tip.sh | 60 ++++++++++ github-action/tests/test-threshold-eval.sh | 111 ++++++++++++++++++ 4 files changed, 303 insertions(+) create mode 100644 .github/workflows/test-github-action.yml create mode 100755 github-action/tests/test-action-yml-defaults.sh create mode 100755 github-action/tests/test-pr-comment-tip.sh create mode 100755 github-action/tests/test-threshold-eval.sh diff --git a/.github/workflows/test-github-action.yml b/.github/workflows/test-github-action.yml new file mode 100644 index 00000000..b7836ee7 --- /dev/null +++ b/.github/workflows/test-github-action.yml @@ -0,0 +1,59 @@ +name: Test github-action/ Composite Action + +# Runs on every PR touching github-action/ (the cloud-API variant) and on +# the same paths landing on main. The Rafter API is NOT exercised — these +# are pure-bash unit tests of the threshold-eval and PR-comment logic +# plus a drift detector on action.yml's load-bearing defaults. +# +# An end-to-end test against the real Rafter API is a future addition +# (would require an injectable RAFTER_API_KEY secret and a fixture repo). +on: + push: + branches: + - main + paths: + - 'github-action/**' + - '.github/workflows/test-github-action.yml' + pull_request: + paths: + - 'github-action/**' + - '.github/workflows/test-github-action.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + test-threshold-evaluation: + name: Threshold-evaluation case statement + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run threshold-eval unit test + run: bash github-action/tests/test-threshold-eval.sh + + test-pr-comment-tip: + name: PR-comment report-only tip + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run PR-comment-tip unit test + run: bash github-action/tests/test-pr-comment-tip.sh + + test-action-yml-defaults: + name: action.yml drift detector + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run action.yml defaults / drift check + run: bash github-action/tests/test-action-yml-defaults.sh + + test-yaml-validity: + name: action.yml is valid YAML + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Validate YAML + run: | + python -c "import yaml; yaml.safe_load(open('github-action/action.yml'))" \ + && echo "OK: github-action/action.yml parses as valid YAML" diff --git a/github-action/tests/test-action-yml-defaults.sh b/github-action/tests/test-action-yml-defaults.sh new file mode 100755 index 00000000..084748e6 --- /dev/null +++ b/github-action/tests/test-action-yml-defaults.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# +# Drift-detection check for github-action/action.yml. Asserts that the +# `severity-threshold` input default is literally 'none' and that the +# PR-comment step still wires SEVERITY_THRESHOLD through as an env var +# (the GitHub-recommended pattern that prevents script injection from +# inputs). +# +# These are the load-bearing properties of the v0.8.3 default flip; if +# someone reverts either, this test fails loudly before the release ships. + +set -eu + +ACTION_YML="$(cd "$(dirname "$0")/.." && pwd)/action.yml" + +if [ ! -f "$ACTION_YML" ]; then + echo "FAIL: $ACTION_YML not found" + exit 1 +fi + +failures=0 + +# 1. severity-threshold default must be the string 'none'. Match the YAML +# block scalar exactly to avoid matching the input description text. +if awk ' + /^[[:space:]]*severity-threshold:/ { in_block=1; next } + in_block && /^[[:space:]]*default:/ { print; exit } +' "$ACTION_YML" | grep -qE "default: *'none'"; then + echo "PASS: severity-threshold default is 'none'" +else + echo "FAIL: severity-threshold default is NOT 'none' in $ACTION_YML" + awk ' + /^[[:space:]]*severity-threshold:/ { in_block=1; next } + in_block && /^[[:space:]]*default:/ { print " found: " $0; exit } + ' "$ACTION_YML" + failures=$((failures+1)) +fi + +# 2. SEVERITY_THRESHOLD must be wired through as an env var on the +# "Comment on PR" step (GitHub-recommended template-injection mitigation). +if grep -qE "SEVERITY_THRESHOLD: *\\\$\\{\\{ *inputs\\.severity-threshold *\\}\\}" "$ACTION_YML"; then + echo "PASS: SEVERITY_THRESHOLD wired as env var (not script interpolation)" +else + echo "FAIL: SEVERITY_THRESHOLD env var passthrough missing from $ACTION_YML" + failures=$((failures+1)) +fi + +# 3. The report-only tip block must be present and gated on both conditions. +if grep -qE '\[ "\$FINDINGS_COUNT" -gt 0 \] && \[ "\$SEVERITY_THRESHOLD" = "none" \]' "$ACTION_YML"; then + echo "PASS: report-only tip block gated on (findings > 0) AND (threshold == 'none')" +else + echo "FAIL: report-only tip block missing or mis-gated in $ACTION_YML" + failures=$((failures+1)) +fi + +# 4. The threshold-eval step must still handle 'none' as a no-op +# (no FAIL=1 in the none branch). +if awk ' + /none\)/ { in_none=1; next } + in_none && /;;/ { in_none=0; next } + in_none { print } +' "$ACTION_YML" | grep -qE "FAIL *= *1"; then + echo "FAIL: 'none' branch of threshold-eval sets FAIL=1 — that would break the default" + failures=$((failures+1)) +else + echo "PASS: 'none' branch of threshold-eval does not set FAIL=1" +fi + +echo "" +echo "── results ───────────────────────────────────────────────────────────" +echo "Failures: $failures" +[ "$failures" -eq 0 ] || exit 1 +echo "OK: action.yml load-bearing properties intact" diff --git a/github-action/tests/test-pr-comment-tip.sh b/github-action/tests/test-pr-comment-tip.sh new file mode 100755 index 00000000..01e8ba68 --- /dev/null +++ b/github-action/tests/test-pr-comment-tip.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# +# Unit test for the new PR-comment "report-only tip" block in +# github-action/action.yml. Re-implements the if block verbatim and +# exercises every input combination. +# +# The tip should appear iff (FINDINGS_COUNT > 0) AND (SEVERITY_THRESHOLD == 'none'). + +set -u + +failures=0 +total=0 + +# Mirror of the new block under the "Comment on PR" step's COMMENT_FILE builder. +emit_tip_if_applicable() { + if [ "$FINDINGS_COUNT" -gt 0 ] && [ "$SEVERITY_THRESHOLD" = "none" ]; then + echo "> :information_source: This run is report-only. To fail the build on critical/high findings, set \`severity-threshold: high\` in your workflow." + echo "" + fi +} + +TIP_NEEDLE="report-only" + +# assert_tip +assert_tip() { + local name="$1"; local expected="$2" + FINDINGS_COUNT="$3"; SEVERITY_THRESHOLD="$4" + total=$((total+1)) + + local out + out=$(emit_tip_if_applicable) + local has_tip="no" + if echo "$out" | grep -q "$TIP_NEEDLE"; then has_tip="yes"; fi + + if [ "$has_tip" != "$expected" ]; then + echo "FAIL: $name — findings=$FINDINGS_COUNT threshold=$SEVERITY_THRESHOLD → expected tip=$expected got $has_tip" + failures=$((failures+1)) + else + echo "PASS: $name (tip=$has_tip)" + fi +} + +echo "── report-only tip should appear ────────────────────────────────────" +assert_tip "findings + none" yes 5 none +assert_tip "single finding + none" yes 1 none + +echo "── report-only tip should NOT appear ────────────────────────────────" +assert_tip "no findings + none" no 0 none +assert_tip "findings + high" no 5 high +assert_tip "findings + critical" no 5 critical +assert_tip "findings + medium" no 5 medium +assert_tip "findings + low" no 5 low +assert_tip "no findings + high" no 0 high +assert_tip "no findings + critical" no 0 critical + +echo "" +echo "── results ───────────────────────────────────────────────────────────" +echo "Total: $total Failures: $failures" +[ "$failures" -eq 0 ] || exit 1 +echo "OK: PR-comment tip appears exactly when (findings > 0) AND (threshold == 'none')" diff --git a/github-action/tests/test-threshold-eval.sh b/github-action/tests/test-threshold-eval.sh new file mode 100755 index 00000000..99dbfc30 --- /dev/null +++ b/github-action/tests/test-threshold-eval.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# +# Unit test for the "Evaluate severity threshold" step in +# github-action/action.yml. Re-implements the case statement verbatim and +# exercises every branch with deliberate inputs. +# +# If you change the case body in action.yml, you MUST change it here too — +# the test-action-yml-defaults check enforces drift detection on the default +# value, but the case body itself is duplicated by design (sourcing bash out +# of YAML at test time is fragile). +# +# Exit 0 = all cases pass. Exit 1 = at least one case failed. + +set -u + +failures=0 +total=0 + +# Mirror of the case body in github-action/action.yml under the +# "Evaluate severity threshold" step. Returns 1 if the threshold would +# fail the build given the current *_COUNT envs, else 0. +evaluate_threshold() { + local FAIL=0 + case "$SEVERITY_THRESHOLD" in + critical) + [ "$CRITICAL_COUNT" -gt 0 ] && FAIL=1 + ;; + high) + [ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] && FAIL=1 + ;; + medium) + [ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] || [ "$MEDIUM_COUNT" -gt 0 ] && FAIL=1 + ;; + low) + [ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] || [ "$MEDIUM_COUNT" -gt 0 ] || [ "$LOW_COUNT" -gt 0 ] && FAIL=1 + ;; + none) + FAIL=0 + ;; + *) + [ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] && FAIL=1 + ;; + esac + return $FAIL +} + +# assert_threshold +assert_threshold() { + local name="$1"; local expected="$2" + SEVERITY_THRESHOLD="$3" + CRITICAL_COUNT="$4"; HIGH_COUNT="$5"; MEDIUM_COUNT="$6"; LOW_COUNT="$7" + total=$((total+1)) + + evaluate_threshold + local actual=$? + + if [ "$actual" != "$expected" ]; then + echo "FAIL: $name — threshold=$SEVERITY_THRESHOLD crit=$CRITICAL_COUNT high=$HIGH_COUNT med=$MEDIUM_COUNT low=$LOW_COUNT → expected exit=$expected got $actual" + failures=$((failures+1)) + else + echo "PASS: $name" + fi +} + +echo "── 'none' threshold (the new default) — must never fail ─────────────" +assert_threshold "none + no findings" 0 none 0 0 0 0 +assert_threshold "none + low only" 0 none 0 0 0 7 +assert_threshold "none + medium only" 0 none 0 0 3 0 +assert_threshold "none + high only" 0 none 0 5 0 0 +assert_threshold "none + critical only" 0 none 2 0 0 0 +assert_threshold "none + everything" 0 none 9 9 9 9 + +echo "── 'critical' threshold — fail only on critical ────────────────────" +assert_threshold "critical + clean" 0 critical 0 0 0 0 +assert_threshold "critical + only high" 0 critical 0 4 0 0 +assert_threshold "critical + only medium" 0 critical 0 0 4 0 +assert_threshold "critical + only low" 0 critical 0 0 0 4 +assert_threshold "critical + critical=1" 1 critical 1 0 0 0 +assert_threshold "critical + critical+high" 1 critical 1 5 0 0 + +echo "── 'high' threshold — fail on critical or high ─────────────────────" +assert_threshold "high + clean" 0 high 0 0 0 0 +assert_threshold "high + only medium" 0 high 0 0 4 0 +assert_threshold "high + only low" 0 high 0 0 0 4 +assert_threshold "high + critical only" 1 high 1 0 0 0 +assert_threshold "high + high only" 1 high 0 1 0 0 +assert_threshold "high + critical+high" 1 high 1 1 0 0 + +echo "── 'medium' threshold — fail on crit/high/medium ───────────────────" +assert_threshold "medium + clean" 0 medium 0 0 0 0 +assert_threshold "medium + only low" 0 medium 0 0 0 4 +assert_threshold "medium + critical only" 1 medium 1 0 0 0 +assert_threshold "medium + high only" 1 medium 0 1 0 0 +assert_threshold "medium + medium only" 1 medium 0 0 1 0 + +echo "── 'low' threshold — fail on anything ──────────────────────────────" +assert_threshold "low + clean" 0 low 0 0 0 0 +assert_threshold "low + only low" 1 low 0 0 0 1 +assert_threshold "low + critical only" 1 low 1 0 0 0 + +echo "── unknown threshold — falls back to 'high' behavior ───────────────" +assert_threshold "unknown + clean" 0 badvalue 0 0 0 0 +assert_threshold "unknown + critical" 1 badvalue 1 0 0 0 +assert_threshold "unknown + high" 1 badvalue 0 1 0 0 +assert_threshold "unknown + medium only" 0 badvalue 0 0 3 0 + +echo "" +echo "── results ───────────────────────────────────────────────────────────" +echo "Total: $total Failures: $failures" +[ "$failures" -eq 0 ] || exit 1 +echo "OK: all threshold-eval branches behave as expected"