Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/test-github-action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Test github-action/ Composite Action

# Runs on every PR touching github-action/ (the cloud-API variant) and on
# the same paths landing on main. The Rafter API is NOT exercised — these
# are pure-bash unit tests of the threshold-eval and PR-comment logic
# plus a drift detector on action.yml's load-bearing defaults.
#
# An end-to-end test against the real Rafter API is a future addition
# (would require an injectable RAFTER_API_KEY secret and a fixture repo).
on:
push:
branches:
- main
paths:
- 'github-action/**'
- '.github/workflows/test-github-action.yml'
pull_request:
paths:
- 'github-action/**'
- '.github/workflows/test-github-action.yml'
workflow_dispatch:

permissions:
contents: read

jobs:
test-threshold-evaluation:
name: Threshold-evaluation case statement
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run threshold-eval unit test
run: bash github-action/tests/test-threshold-eval.sh

test-pr-comment-tip:
name: PR-comment report-only tip
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run PR-comment-tip unit test
run: bash github-action/tests/test-pr-comment-tip.sh

test-action-yml-defaults:
name: action.yml drift detector
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run action.yml defaults / drift check
run: bash github-action/tests/test-action-yml-defaults.sh

test-yaml-validity:
name: action.yml is valid YAML
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate YAML
run: |
python -c "import yaml; yaml.safe_load(open('github-action/action.yml'))" \
&& echo "OK: github-action/action.yml parses as valid YAML"
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.8.3] - 2026-05-31

### Changed
- **`github-action/` cloud variant: `severity-threshold` default is now `none` (report-only)** (#148). Previously a fresh install with no `severity-threshold` set would fail the build on any critical/high finding — so first-install on a real repo almost always broke CI on day one, the worst possible first impression for a security tool. The default now matches Snyk / CodeQL / Semgrep: scan completes, findings post as a PR comment, SARIF uploads to the Security tab, but the workflow exits 0 regardless of severity. When findings exist under the report-only default, the PR comment includes a one-line tip pointing to `severity-threshold: high` for opt-in enforcement. Workflows that set `severity-threshold` explicitly are unaffected. README updated with an "Enforce in CI (recommended after first scan)" example. Affects only the cloud-API action at `Raftersecurity/rafter-cli/github-action@…`; the root `Raftersecurity/rafter-cli@v…` local-secrets action is unchanged.

### Infrastructure
- `publish.yaml` migrated to npm Trusted Publishing (OIDC) — no more long-lived `NPM_TOKEN` (#142). Renamed `publish.yml` → `publish.yaml` to match npm's Trusted Publisher manifest, and added `--skip-existing` on the PyPI upload so retries of a partially-failed release no longer error on "File already exists" (#143).
- `node/package.json` now declares `repository` / `homepage` / `bugs` / `license` / `author` metadata so the npm listing renders sidebar links correctly (#145).
- Cursor recipe in the rafter SKILL `Setup` section now leads with `--local` for ephemeral / containerized setups (#147).

## [0.8.2] - 2026-05-26

### Changed
Expand Down
73 changes: 73 additions & 0 deletions github-action/tests/test-action-yml-defaults.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
#
# Drift-detection check for github-action/action.yml. Asserts that the
# `severity-threshold` input default is literally 'none' and that the
# PR-comment step still wires SEVERITY_THRESHOLD through as an env var
# (the GitHub-recommended pattern that prevents script injection from
# inputs).
#
# These are the load-bearing properties of the v0.8.3 default flip; if
# someone reverts either, this test fails loudly before the release ships.

set -eu

ACTION_YML="$(cd "$(dirname "$0")/.." && pwd)/action.yml"

if [ ! -f "$ACTION_YML" ]; then
echo "FAIL: $ACTION_YML not found"
exit 1
fi

failures=0

# 1. severity-threshold default must be the string 'none'. Match the YAML
# block scalar exactly to avoid matching the input description text.
if awk '
/^[[:space:]]*severity-threshold:/ { in_block=1; next }
in_block && /^[[:space:]]*default:/ { print; exit }
' "$ACTION_YML" | grep -qE "default: *'none'"; then
echo "PASS: severity-threshold default is 'none'"
else
echo "FAIL: severity-threshold default is NOT 'none' in $ACTION_YML"
awk '
/^[[:space:]]*severity-threshold:/ { in_block=1; next }
in_block && /^[[:space:]]*default:/ { print " found: " $0; exit }
' "$ACTION_YML"
failures=$((failures+1))
fi

# 2. SEVERITY_THRESHOLD must be wired through as an env var on the
# "Comment on PR" step (GitHub-recommended template-injection mitigation).
if grep -qE "SEVERITY_THRESHOLD: *\\\$\\{\\{ *inputs\\.severity-threshold *\\}\\}" "$ACTION_YML"; then
echo "PASS: SEVERITY_THRESHOLD wired as env var (not script interpolation)"
else
echo "FAIL: SEVERITY_THRESHOLD env var passthrough missing from $ACTION_YML"
failures=$((failures+1))
fi

# 3. The report-only tip block must be present and gated on both conditions.
if grep -qE '\[ "\$FINDINGS_COUNT" -gt 0 \] && \[ "\$SEVERITY_THRESHOLD" = "none" \]' "$ACTION_YML"; then
echo "PASS: report-only tip block gated on (findings > 0) AND (threshold == 'none')"
else
echo "FAIL: report-only tip block missing or mis-gated in $ACTION_YML"
failures=$((failures+1))
fi

# 4. The threshold-eval step must still handle 'none' as a no-op
# (no FAIL=1 in the none branch).
if awk '
/none\)/ { in_none=1; next }
in_none && /;;/ { in_none=0; next }
in_none { print }
' "$ACTION_YML" | grep -qE "FAIL *= *1"; then
echo "FAIL: 'none' branch of threshold-eval sets FAIL=1 — that would break the default"
failures=$((failures+1))
else
echo "PASS: 'none' branch of threshold-eval does not set FAIL=1"
fi

echo ""
echo "── results ───────────────────────────────────────────────────────────"
echo "Failures: $failures"
[ "$failures" -eq 0 ] || exit 1
echo "OK: action.yml load-bearing properties intact"
60 changes: 60 additions & 0 deletions github-action/tests/test-pr-comment-tip.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
#
# Unit test for the new PR-comment "report-only tip" block in
# github-action/action.yml. Re-implements the if block verbatim and
# exercises every input combination.
#
# The tip should appear iff (FINDINGS_COUNT > 0) AND (SEVERITY_THRESHOLD == 'none').

set -u

failures=0
total=0

# Mirror of the new block under the "Comment on PR" step's COMMENT_FILE builder.
emit_tip_if_applicable() {
if [ "$FINDINGS_COUNT" -gt 0 ] && [ "$SEVERITY_THRESHOLD" = "none" ]; then
echo "> :information_source: This run is report-only. To fail the build on critical/high findings, set \`severity-threshold: high\` in your workflow."
echo ""
fi
}

TIP_NEEDLE="report-only"

# assert_tip <name> <expected: yes|no> <findings> <threshold>
assert_tip() {
local name="$1"; local expected="$2"
FINDINGS_COUNT="$3"; SEVERITY_THRESHOLD="$4"
total=$((total+1))

local out
out=$(emit_tip_if_applicable)
local has_tip="no"
if echo "$out" | grep -q "$TIP_NEEDLE"; then has_tip="yes"; fi

if [ "$has_tip" != "$expected" ]; then
echo "FAIL: $name — findings=$FINDINGS_COUNT threshold=$SEVERITY_THRESHOLD → expected tip=$expected got $has_tip"
failures=$((failures+1))
else
echo "PASS: $name (tip=$has_tip)"
fi
}

echo "── report-only tip should appear ────────────────────────────────────"
assert_tip "findings + none" yes 5 none
assert_tip "single finding + none" yes 1 none

echo "── report-only tip should NOT appear ────────────────────────────────"
assert_tip "no findings + none" no 0 none
assert_tip "findings + high" no 5 high
assert_tip "findings + critical" no 5 critical
assert_tip "findings + medium" no 5 medium
assert_tip "findings + low" no 5 low
assert_tip "no findings + high" no 0 high
assert_tip "no findings + critical" no 0 critical

echo ""
echo "── results ───────────────────────────────────────────────────────────"
echo "Total: $total Failures: $failures"
[ "$failures" -eq 0 ] || exit 1
echo "OK: PR-comment tip appears exactly when (findings > 0) AND (threshold == 'none')"
111 changes: 111 additions & 0 deletions github-action/tests/test-threshold-eval.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
#!/usr/bin/env bash
#
# Unit test for the "Evaluate severity threshold" step in
# github-action/action.yml. Re-implements the case statement verbatim and
# exercises every branch with deliberate inputs.
#
# If you change the case body in action.yml, you MUST change it here too —
# the test-action-yml-defaults check enforces drift detection on the default
# value, but the case body itself is duplicated by design (sourcing bash out
# of YAML at test time is fragile).
#
# Exit 0 = all cases pass. Exit 1 = at least one case failed.

set -u

failures=0
total=0

# Mirror of the case body in github-action/action.yml under the
# "Evaluate severity threshold" step. Returns 1 if the threshold would
# fail the build given the current *_COUNT envs, else 0.
evaluate_threshold() {
local FAIL=0
case "$SEVERITY_THRESHOLD" in
critical)
[ "$CRITICAL_COUNT" -gt 0 ] && FAIL=1
;;
high)
[ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] && FAIL=1
;;
medium)
[ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] || [ "$MEDIUM_COUNT" -gt 0 ] && FAIL=1
;;
low)
[ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] || [ "$MEDIUM_COUNT" -gt 0 ] || [ "$LOW_COUNT" -gt 0 ] && FAIL=1
;;
none)
FAIL=0
;;
*)
[ "$CRITICAL_COUNT" -gt 0 ] || [ "$HIGH_COUNT" -gt 0 ] && FAIL=1
;;
esac
return $FAIL
}

# assert_threshold <name> <expected_exit> <severity> <crit> <high> <med> <low>
assert_threshold() {
local name="$1"; local expected="$2"
SEVERITY_THRESHOLD="$3"
CRITICAL_COUNT="$4"; HIGH_COUNT="$5"; MEDIUM_COUNT="$6"; LOW_COUNT="$7"
total=$((total+1))

evaluate_threshold
local actual=$?

if [ "$actual" != "$expected" ]; then
echo "FAIL: $name — threshold=$SEVERITY_THRESHOLD crit=$CRITICAL_COUNT high=$HIGH_COUNT med=$MEDIUM_COUNT low=$LOW_COUNT → expected exit=$expected got $actual"
failures=$((failures+1))
else
echo "PASS: $name"
fi
}

echo "── 'none' threshold (the new default) — must never fail ─────────────"
assert_threshold "none + no findings" 0 none 0 0 0 0
assert_threshold "none + low only" 0 none 0 0 0 7
assert_threshold "none + medium only" 0 none 0 0 3 0
assert_threshold "none + high only" 0 none 0 5 0 0
assert_threshold "none + critical only" 0 none 2 0 0 0
assert_threshold "none + everything" 0 none 9 9 9 9

echo "── 'critical' threshold — fail only on critical ────────────────────"
assert_threshold "critical + clean" 0 critical 0 0 0 0
assert_threshold "critical + only high" 0 critical 0 4 0 0
assert_threshold "critical + only medium" 0 critical 0 0 4 0
assert_threshold "critical + only low" 0 critical 0 0 0 4
assert_threshold "critical + critical=1" 1 critical 1 0 0 0
assert_threshold "critical + critical+high" 1 critical 1 5 0 0

echo "── 'high' threshold — fail on critical or high ─────────────────────"
assert_threshold "high + clean" 0 high 0 0 0 0
assert_threshold "high + only medium" 0 high 0 0 4 0
assert_threshold "high + only low" 0 high 0 0 0 4
assert_threshold "high + critical only" 1 high 1 0 0 0
assert_threshold "high + high only" 1 high 0 1 0 0
assert_threshold "high + critical+high" 1 high 1 1 0 0

echo "── 'medium' threshold — fail on crit/high/medium ───────────────────"
assert_threshold "medium + clean" 0 medium 0 0 0 0
assert_threshold "medium + only low" 0 medium 0 0 0 4
assert_threshold "medium + critical only" 1 medium 1 0 0 0
assert_threshold "medium + high only" 1 medium 0 1 0 0
assert_threshold "medium + medium only" 1 medium 0 0 1 0

echo "── 'low' threshold — fail on anything ──────────────────────────────"
assert_threshold "low + clean" 0 low 0 0 0 0
assert_threshold "low + only low" 1 low 0 0 0 1
assert_threshold "low + critical only" 1 low 1 0 0 0

echo "── unknown threshold — falls back to 'high' behavior ───────────────"
assert_threshold "unknown + clean" 0 badvalue 0 0 0 0
assert_threshold "unknown + critical" 1 badvalue 1 0 0 0
assert_threshold "unknown + high" 1 badvalue 0 1 0 0
assert_threshold "unknown + medium only" 0 badvalue 0 0 3 0

echo ""
echo "── results ───────────────────────────────────────────────────────────"
echo "Total: $total Failures: $failures"
[ "$failures" -eq 0 ] || exit 1
echo "OK: all threshold-eval branches behave as expected"
2 changes: 1 addition & 1 deletion node/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@rafter-security/cli",
"version": "0.8.2",
"version": "0.8.3",
"type": "module",
"repository": {
"type": "git",
Expand Down
2 changes: 1 addition & 1 deletion node/resources/rafter-security-skill.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: rafter-security
description: Security toolkit for AI workflows. Use when scanning code or repos for vulnerabilities, auditing third-party skills/MCPs/agent configs before installing, evaluating shell commands before running them, or generating secure design questions for new features. Provides `rafter run` (remote SAST + SCA, needs RAFTER_API_KEY), `rafter secrets` (offline secrets-only), `rafter agent exec --dry-run` (command-risk classification), and `rafter skill review`.
version: 0.8.2
version: 0.8.3
homepage: https://rafter.so
metadata:
openclaw:
Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "rafter-cli"
version = "0.8.2"
version = "0.8.3"
description = "Rafter CLI — the default security agent for AI workflows. Free for individuals and open source."
authors = ["Rafter Team <hello@rafter.so>"]
license = "MIT"
Expand Down
2 changes: 1 addition & 1 deletion python/rafter_cli/resources/rafter-security-skill.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: rafter-security
description: Security toolkit for AI workflows. Use when scanning code or repos for vulnerabilities, auditing third-party skills/MCPs/agent configs before installing, evaluating shell commands before running them, or generating secure design questions for new features. Provides `rafter run` (remote SAST + SCA, needs RAFTER_API_KEY), `rafter secrets` (offline secrets-only), `rafter agent exec --dry-run` (command-risk classification), and `rafter skill review`.
version: 0.8.2
version: 0.8.3
homepage: https://rafter.so
metadata:
openclaw:
Expand Down
Loading