feat(secrets): gitleaks subprocess backend with regex fallback (closes #159)

[Wolfvin]

Closes #159

Summary

Upgrades codelens secrets to use gitleaks as the primary backend when available (600+ maintained rules, entropy scoring, structured JSON output). Falls back to the existing regex scanner when gitleaks is not installed or --no-gitleaks is set. Gitleaks is an opt-in upgrade, never a hard dependency.

What changed

New files (2)

• scripts/gitleaks_backend.py — self-contained gitleaks integration (454 lines)
  • _gitleaks_available() — detects binary via gitleaks version
  • _run_gitleaks() — invokes gitleaks detect --no-git --report-format json, writes to temp file, parses JSON, handles both list and {Results:[...]} schemas (older gitleaks versions), cleans up temp file
  • _normalize_gitleaks_findings() — maps gitleaks JSON fields to CodeLens finding schema (RuleID→type, File→file, StartLine→line, Secret→masked match/value, Tags→tags, Fingerprint→fingerprint)
  • _infer_severity() — heuristic severity mapping (aws-access-key/private-key/github-pat/stripe-secret/etc → critical; 'critical'/'high'/'medium'/'low' in rule ID or tags → matching severity; default → high)
  • _mask_secret() — first 4 chars + *** (matches secrets_engine._mask_value convention)
  • scan_with_gitleaks() — public entry point; returns CodeLens-shaped result dict with backend='gitleaks', or None if gitleaks unavailable
  • GitleaksError — caught by caller, falls back to regex with warning
• tests/test_secrets_gitleaks.py — 56 tests (543 lines)

Modified files (2)

• scripts/commands/secrets.py (73 lines added):
  • Add --no-gitleaks flag (force regex backend)
  • execute() tries gitleaks first (unless --no-gitleaks); on gitleaks failure, falls back to regex with stderr warning (never crashes)
  • Tags result with backend='regex' or 'gitleaks'
  • When gitleaks unavailable, adds gitleaks_hint field with install URL
  • stats.backend also set so compact/ai formatters pick it up
• SKILL.md — add "Gitleaks-Backed Secrets Scanner" to v8.2 capability pillars with install instructions

Acceptance criteria (from issue body)

• _gitleaks_available() detection in scripts/commands/secrets.py — in gitleaks_backend.py (called from commands/secrets.py)
• Gitleaks subprocess invocation with JSON output mode — _run_gitleaks() uses --report-format json --report-path <tempfile>
• Result normalizer: gitleaks JSON → CodeLens findings schema — _normalize_gitleaks_findings()
• Graceful fallback to existing regex scanner with warning message — execute() catches GitleaksError, prints stderr warning, runs detect_secrets()
• --no-gitleaks flag to force regex fallback (useful for testing) — added to add_args()
• stats.backend field in output: "gitleaks" or "regex" — set in both scan_with_gitleaks() and execute() regex path
• Installation hint in output when gitleaks not found — gitleaks_hint field with install URL
• SKILL.md updated: note that gitleaks improves accuracy, link to install docs — added as capability pillar [ARCH] Replace flat registry with true graph data model (nodes + edges) #8
• Existing tests remain green (mock gitleaks subprocess in tests) — 56 new tests, 0 regressions
• sync_command_count.py --apply run (count unchanged — same command, upgraded backend) — verified count = 70

Design decisions

1. gitleaks logic in NEW module (gitleaks_backend.py), not in secrets_engine.py — avoids collision with PR feat(agent-ux): confidence scores per finding + schema_version in JSON output (closes #5) #148 (which touches secrets_engine.py for confidence scores). commands/secrets.py is the wiring point; secrets_engine.py is untouched.

2. Gitleaks is OPT-IN — if not installed, regex runs unchanged with a hint. Never a hard dependency. This matches the issue's explicit requirement: "Some environments cannot install binaries (locked-down CI, read-only filesystems). The regex fallback ensures codelens secrets always works, everywhere, without setup."

3. --no-git flag passed to gitleaks — scans the working tree only (not git history) to match the regex backend's behavior. Git history scanning can be added later via a --git-history flag if needed. This keeps the two backends semantically equivalent.

4. Normalized findings include backend field — so consumers can tell which scanner produced each finding (useful for debugging false positives/negatives, and for future mixed-backend scenarios).

5. Severity inference is heuristic — gitleaks doesn't have a native severity field. The heuristic maps high-value secret types (aws-access-key, private-key, github-pat, etc.) to critical, checks for severity keywords in rule ID/tags, and defaults to high (gitleaks rules are curated, so default-high is reasonable). Documented in the _infer_severity docstring.

6. GitleaksError caught, never crashes — the command always returns a result. Gitleaks failure produces a stderr warning and falls back to regex. This matches the issue's "graceful fallback" requirement.

Tests

• New: 56 passed in tests/test_secrets_gitleaks.py
  • TestGitleaksAvailable (5) — detection: available, not-found, timeout, nonzero, exception
  • TestGitleaksVersion (2) — version string / None
  • TestMaskSecret (5) — long, short, empty, 4-char, 5-char
  • TestInferSeverity (12) — all high-value markers, tag-based, rule-ID-based, default
  • TestNormalizeFindings (7) — basic, masking, abs→rel paths, tags-as-string, empty, non-dict, missing fields
  • TestStatsRiskRecs (9) — counts, files_with_findings, risk cascade, recommendation content
  • TestRunGitleaks (7) — JSON parse, empty file, missing file, nonzero exit, timeout, invalid JSON, {Results:[...]} schema
  • TestScanWithGitleaks (4) — returns None when unavailable, full result dict, severity filter, nonexistent workspace
  • TestCliIntegration (4) — --no-gitleaks in help, regex backend + hint when gitleaks absent, --no-gitleaks suppresses hint, stats.backend set
• Regression check: test_secrets_engine + test_cli + test_command_count + test_command_registry → 53 passed, 0 failed (no regressions)
• Command count: unchanged at 70 (same command, upgraded backend)

End-to-end verification

$ python3 scripts/codelens.py secrets --help | grep no-gitleaks
                           [--max-files MAX_FILES] [--no-gitleaks]

$ python3 scripts/codelens.py secrets tests/fixtures  # gitleaks not installed
{
  "status": "ok",
  "backend": "regex",
  "gitleaks_hint": "gitleaks not found — using built-in regex scanner (lower
    accuracy). Install gitleaks for 600+ maintained rules and entropy scoring:
    https://github.com/gitleaks/gitleaks",
  "stats": {"backend": "regex", ...},
  "findings": [...]
}

$ python3 scripts/codelens.py secrets tests/fixtures --no-gitleaks
{
  "status": "ok",
  "backend": "regex",
  # gitleaks_hint NOT present (user explicitly opted out)
  ...
}

Findings (per pre-flight SKILL.md — flag to BOS)

• PR feat(agent-ux): confidence scores per finding + schema_version in JSON output (closes #5) #148 collision check: PR feat(agent-ux): confidence scores per finding + schema_version in JSON output (closes #5) #148 (issue Agent UX: Confidence scores, delta output, and dependency graph between findings #5, open) touches scripts/secrets_engine.py (+16 lines, adding confidence scores via scripts/confidence.py). This PR touches scripts/commands/secrets.py (different file) and adds scripts/gitleaks_backend.py (new file). No file collision — both can merge independently. The gitleaks backend's normalized findings include the same schema fields (type, file, line, match, severity, category) that PR feat(agent-ux): confidence scores per finding + schema_version in JSON output (closes #5) #148's confidence.py expects, so confidence scoring will work on gitleaks findings the same way as regex findings.

Branch

feat/issue-159-secrets-gitleaks-backend

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud