fix: redact finding description, remediation and proof before DB insert

[Srejoye]

Problem

executor.py called redact(output) on the raw CLI text file (line 302) but the parsed findings written to the findings table were never redacted:

# _upsert_findings_and_report — before this fix
finding["description"],          # ← no redact()
finding.get("remediation", ""),  # ← no redact()
finding.get("proof"),            # ← no redact()

reporting.py::_normalize_finding() does call redact() — but only at PDF/HTML export time. The DB row, the /api/v1/findings JSON API, SARIF exports, CSV exports, and any direct DB reads all returned raw unredacted content.

redact_dict() already existed in redaction.py for exactly this purpose — it was just never called before storage.

Fix

backend/secuscan/executor.py

1. Add redact_dict to the import from .redaction
2. Call finding = redact_dict(finding) before the INSERT in both _upsert_findings_and_report and _upsert_findings_and_report_from_scanner

# after fix — both INSERT loops
for finding in findings_data:
    finding = redact_dict(finding)   # redact secrets before storage
    ...INSERT...

Tests

New file testing/backend/unit/test_findings_redaction.py — 6 tests:

Test	What it verifies
test_redact_dict_redacts_aws_key_in_description	Secret in description/remediation is replaced with [REDACTED]
test_redact_dict_leaves_clean_finding_unchanged	Clean findings pass through unmodified
test_redact_dict_handles_nested_metadata	Nested metadata dict walked recursively; non-strings untouched
test_redact_dict_handles_none_proof	None proof does not raise
test_redact_dict_handles_missing_keys_gracefully	Minimal findings with no description/remediation work fine
test_upsert_findings_redacts_description_before_insert	Integration: DB row verified to not contain raw secret after INSERT

No breaking changes

redact_dict uses existing infrastructure. Non-secret content passes through unchanged. No API or schema changes.

Closes #306

[utksh1]

Thanks — this is close, but I need changes before merge. The findings rows are redacted, but the same parsed findings are still written to tasks.structured_json before redaction, so secrets can still be persisted and exposed through task-result paths. Please redact the parsed structure before any DB write, keep findings/report persistence using that redacted copy, and add a regression that verifies structured_json does not contain the secret either.

[utksh1]

Thanks for following up. Clarifying the change request so it is actionable:

Why this is blocked:
Thanks — this is close, but I need changes before merge. The findings rows are redacted, but the same parsed findings are still written to tasks.structured_json before redaction, so secrets can still be persisted and exposed through task-result paths. Please redact the parsed structure before any DB write, keep findings/report persistence using that redacted copy, and add a regression that verifies structured_json does not contain the secret either.

What to do next:

• Fix the specific issues called out above.
• Push the updated branch and make sure the relevant CI checks pass.
• Reply here when ready for re-review.

[Srejoye]

Hi @utksh1, I've addressed the feedback. The fix now redacts all findings before any DB write — parsed["findings"] is replaced with a fully redacted list before the structured_json write, so both tasks.structured_json and the findings table rows use the clean copy. Added a regression assertion to the integration test that verifies structured_json does not contain the secret either. All 6 CI checks are passing. Could you please take another look? Thanks!

[utksh1]

Re-reviewed latest state. The branch is now conflicting, and the earlier data-leak blocker still needs verification against current main: redact the parsed structure before any structured_json/write path, not only findings rows, and keep a regression proving structured_json never contains the secret.

[Srejoye]

Donee

[utksh1]

Re-reviewed after the latest push. Still blocked: redaction must happen before every persistence path, including tasks.structured_json, not only the findings rows. Please add a regression proving structured_json never contains the secret.

[Srejoye]

Hi @utksh1, I've rebased cleanly onto current main. The fix is working correctly on both persistence paths:

In executor.py — redacted_findings is built first, parsed["findings"] is replaced with the redacted list, then json.dumps(parsed) is written to structured_json. So the redacted copy reaches both tasks.structured_json and the findings table rows.

The regression is in test_findings_redaction.py — after calling _upsert_findings_and_report, the test fetches tasks.structured_json, parses it, and asserts the secret is absent from findings_in_structured[0]["description"]. Both assertions (findings table row + structured_json) are present and passing.

All CI checks were green on the previous push. Please take another look. Thanks!

[utksh1]

Re-reviewed after the latest push. Still blocked: redaction must happen before every DB persistence path, including tasks.structured_json. Please add a regression proving structured_json never contains the secret, not only findings rows.