From 0d609e0702aadfe16c010ba8b88e4e46a4e80a90 Mon Sep 17 00:00:00 2001 From: jddark62 Date: Sat, 6 Jun 2026 23:53:41 +0530 Subject: [PATCH] Add PIR remediation verification fixtures --- .../post-incident-review/SKILL.md | 68 +++++++-- .../remediation-verification-edge-cases.md | 131 ++++++++++++++++++ 2 files changed, 190 insertions(+), 9 deletions(-) create mode 100644 skills/incident-response/post-incident-review/tests/remediation-verification-edge-cases.md diff --git a/skills/incident-response/post-incident-review/SKILL.md b/skills/incident-response/post-incident-review/SKILL.md index 748fb990..73dbc173 100644 --- a/skills/incident-response/post-incident-review/SKILL.md +++ b/skills/incident-response/post-incident-review/SKILL.md @@ -13,7 +13,7 @@ phase: [recover] frameworks: [NIST-SP-800-61r2] difficulty: beginner time_estimate: "30-60min" -version: "1.0.0" +version: "1.0.1" author: unitoneai license: MIT allowed-tools: Read, Grep, Glob @@ -58,6 +58,7 @@ Before conducting the PIR, gather or confirm: - [ ] **Existing controls** -- Documentation of security controls that were in place at the time of the incident (detection rules, access controls, network segmentation, patching cadence). - [ ] **Previous PIR reports** -- Any prior post-incident reviews for similar incident types, to identify recurring patterns. - [ ] **Metrics data** -- Timestamps needed to compute MTTD, MTTR, and MTTC (see Step 4). +- [ ] **Remediation verification inputs** -- Action tracker, closure criteria, retest evidence, verifier signoff, residual-risk decisions, and recurrence signals for this incident class. --- @@ -266,10 +267,37 @@ Convert analysis findings into specific, measurable, assignable, and time-bound **Remediation action template:** -| ID | Finding | Action | Owner | Priority | Deadline | Tracking | -|---|---|---|---|---|---|---| -| REM-001 | [Specific finding from RCA or control failure mapping] | [Specific remediation action] | [Name and team] | [P0/P1/P2/P3] | [YYYY-MM-DD] | [Ticket ID] | -| REM-002 | [Finding] | [Action] | [Owner] | [Priority] | [Deadline] | [Ticket ID] | +| ID | Finding | Action | Owner | Priority | Deadline | Tracking | Closure Criterion | Verification Method | +|---|---|---|---|---|---|---|---|---| +| REM-001 | [Specific finding from RCA or control failure mapping] | [Specific remediation action] | [Name and team] | [P0/P1/P2/P3] | [YYYY-MM-DD] | [Ticket ID] | [Observable condition that proves the failure mode is addressed] | [Retest, replay, sample, drill, or review] | +| REM-002 | [Finding] | [Action] | [Owner] | [Priority] | [Deadline] | [Ticket ID] | [Criterion] | [Method] | + +Do not mark P0/P1 actions or repeated-control-failure actions complete based only on ticket closure. They need verification evidence against the original failure mode or explicit residual-risk acceptance. + +#### 6.1 Remediation Verification and Recurrence Gate + +For each high-impact remediation action, collect: + +| Field | Required Evidence | +|---|---| +| Action ID | REM-### from the action register | +| Mapped control failure | Preventive, detective, corrective, process, communication, data, or vendor failure | +| Closure criterion | Specific observable state that proves the failure mode is addressed | +| Verification method | Detection replay, segmentation validation, access review sample, patch/config proof, restore drill, tabletop, alert-routing test, vendor evidence, or manual review | +| Verification evidence | Ticket, log export, test output, query ID, change record, screenshot, artifact hash, or report ID | +| Verifier | Person/team independent enough to validate the fix | +| Residual risk status | Fixed, mitigated, accepted, transferred, blocked, or not verified | +| Follow-up review date | Date to recheck high-impact actions and recurrence signals | + +Track recurrence separately: + +| Field | Required Evidence | +|---|---| +| Similar prior incidents | Incident IDs, near misses, or PIRs with matching root cause/control failure | +| Prior action outcome | Verified, incomplete, failed verification, accepted risk, or did not cover this scenario | +| Recurrence signal | Detection name, metric, precursor event, control exception, vulnerability class, or service/path | +| Monitoring window | 30/60/90 days or longer for low-frequency controls | +| Reopen threshold | Count, severity, or condition that reopens the PIR or creates a new action | **Remediation prioritization:** @@ -354,12 +382,24 @@ root cause, and the number/priority of remediation actions identified.] - [Gap or failure identified during retrospective] ### Remediation Plan -| ID | Finding | Action | Owner | Priority | Deadline | Ticket | +| ID | Finding | Action | Owner | Priority | Deadline | Ticket | Closure Criterion | Verification Method | +|---|---|---|---|---|---|---|---|---| +| REM-001 | [Finding] | [Action] | [Owner] | [P0-P3] | [Date] | [ID] | [Evidence that proves failure mode is addressed] | [Retest/replay/sample/drill/review] | + +### Remediation Verification Register +| Action ID | Mapped Control Failure | Verification Method | Verification Evidence | Verifier | Residual Risk Status | Follow-Up Review Date | |---|---|---|---|---|---|---| -| REM-001 | [Finding] | [Action] | [Owner] | [P0-P3] | [Date] | [ID] | +| REM-001 | [Control failure] | [Detection replay / segmentation test / access review sample / tabletop / restore drill / vendor evidence] | [Ticket/log/export/test artifact] | [Name/team] | [Fixed/Mitigated/Accepted/Blocked/Not verified] | [YYYY-MM-DD] | + +### Recurrence Tracking +| Similar Prior Incident | Prior Action Outcome | Recurrence Signal | Monitoring Window | Reopen Threshold | Follow-Up Outcome | +|---|---|---|---|---|---| +| [IR-ID / near miss / none found] | [Verified/Incomplete/Failed verification/Accepted risk/Did not cover scenario] | [Detection/metric/precursor/control exception] | [30/60/90 days] | [condition] | [No recurrence/Recurrence detected/Action reopened/Risk accepted] | ### Follow-Up Schedule - **Remediation Review Date:** [YYYY-MM-DD -- typically 30 days after PIR] +- **Verification Review Date:** [YYYY-MM-DD -- for P0/P1 and repeated control failures] +- **Recurrence Review Date:** [YYYY-MM-DD -- end of monitoring window] - **PIR Report Distribution:** [List of recipients] - **Playbook Updates Required:** [Yes/No -- list specific playbooks] - **Detection Rule Updates Required:** [Yes/No -- list specific rules] @@ -408,9 +448,13 @@ The most common pitfall is skipping the post-incident review entirely, especiall When the PIR focuses on who made mistakes rather than what systemic conditions enabled the incident, participants become defensive, withhold information, and the organization learns nothing. The "lesson learned" becomes "person X should have done Y" rather than "process Z should be changed to prevent this class of error." Enforce blameless ground rules at the start of every PIR and redirect blame-oriented statements to system-level observations. -### Pitfall 3: Identifying Remediation Actions Without Tracking Them +### Pitfall 3: Identifying Remediation Actions Without Verifying Them + +Documenting lessons learned and remediation actions in a PIR report that is then filed and forgotten produces zero security improvement. Every remediation action must be entered into the organization's work tracking system (Jira, ServiceNow, Azure DevOps) with an owner, priority, deadline, closure criterion, verification method, and scheduled review date. The PIR facilitator should schedule a follow-up review (typically 30 days after the PIR) to verify remediation progress and a verification review for high-impact actions. + +### Pitfall 3a: Treating Implemented as Verified -Documenting lessons learned and remediation actions in a PIR report that is then filed and forgotten produces zero security improvement. Every remediation action must be entered into the organization's work tracking system (Jira, ServiceNow, Azure DevOps) with an owner, priority, deadline, and scheduled review date. The PIR facilitator should schedule a follow-up review (typically 30 days after the PIR) to verify remediation progress. +"Implemented" means a team reports that a fix exists. "Verified" means independent evidence proves the fix addresses the original failure mode. A detection rule that has never been replayed, a segmentation change that has not been tested from the relevant source, or a playbook update that responders have not exercised should not close a high-impact PIR action. ### Pitfall 4: Stopping Root Cause Analysis at the Proximate Cause @@ -445,3 +489,9 @@ This skill processes incident response data including timelines, forensic findin 7. **SANS Incident Handler's Handbook -- Lessons Learned Phase** -- https://www.sans.org/white-papers/33901/ 8. **ISO/IEC 27035-2:2023** -- Information Security Incident Management -- Part 2: Guidelines to Plan and Prepare for Incident Response -- https://www.iso.org/standard/78974.html 9. **VERIS (Vocabulary for Event Recording and Incident Sharing)** -- http://veriscommunity.net/ + +--- + +## 10. Changelog + +- **1.0.1** -- Added remediation verification, residual-risk, and recurrence tracking gates with calibration fixtures. diff --git a/skills/incident-response/post-incident-review/tests/remediation-verification-edge-cases.md b/skills/incident-response/post-incident-review/tests/remediation-verification-edge-cases.md new file mode 100644 index 00000000..5208edf7 --- /dev/null +++ b/skills/incident-response/post-incident-review/tests/remediation-verification-edge-cases.md @@ -0,0 +1,131 @@ +# Remediation Verification Edge Cases + +These fixtures verify that `post-incident-review` separates action creation, implementation, verification, residual-risk acceptance, and recurrence tracking. + +```yaml +case_id: PIR-VERIFY-01 +title: Detection rule action is implemented but not replay-tested +action: + id: REM-001 + mapped_control_failure: detective detection gap + priority: P1 + closure_criterion: alert fires and routes to on-call queue for representative telemetry + implementation_ticket: closed + verification_method: detection replay + verification_evidence: missing +expected_classification: + status: Not verified + reason: "Ticket closure does not prove the detection catches the original failure mode." +``` + +```yaml +case_id: PIR-VERIFY-02 +title: Detection remediation verified with replay and routing evidence +action: + id: REM-002 + mapped_control_failure: detective alert routing gap + priority: P1 + closure_criterion: replayed telemetry creates high-severity alert and pages SOC + verification_method: detection replay + verification_evidence: + replay_job_id: replay-2026-06-06-01 + alert_id: ALRT-4481 + on_call_page: delivered + verifier: detection-engineering +expected_classification: + status: Verified + reason: "Replay evidence proves the updated detection and routing path work." +``` + +```yaml +case_id: PIR-VERIFY-03 +title: Segmentation fix needs source-to-target validation +action: + id: REM-003 + mapped_control_failure: preventive segmentation failure + priority: P0 + closure_criterion: lateral movement path from user subnet to database subnet is denied and logged + implementation_ticket: firewall rule deployed + verification_method: segmentation validation + verification_evidence: + denied_path_test: missing + allowed_business_path_test: missing +expected_classification: + status: Not verified + reason: "A deployed firewall rule must be tested from the relevant source and destination paths." +``` + +```yaml +case_id: PIR-VERIFY-04 +title: Vendor-dependent action is blocked with interim controls and risk owner +action: + id: REM-004 + mapped_control_failure: vendor patch dependency + priority: P1 + closure_criterion: vendor patch deployed or compensating control verified + vendor_ticket: VEND-9912 + interim_controls: + - WAF virtual patch + - increased logging + residual_risk: + status: Accepted + owner: security-risk-committee + expiry: "2026-07-31" +expected_classification: + status: Accepted risk + reason: "Blocked vendor action has interim controls, accountable risk owner, and expiry date." +``` + +```yaml +case_id: PIR-VERIFY-05 +title: Restore drill verifies backup remediation +action: + id: REM-005 + mapped_control_failure: corrective recovery gap + priority: P1 + closure_criterion: critical database restores within four-hour objective + verification_method: restore drill + verification_evidence: + drill_id: DRILL-2026-06 + restore_time_minutes: 132 + data_integrity_check: passed + verifier: sre-platform +expected_classification: + status: Verified + reason: "Restore drill evidence proves the corrective control meets the recovery objective." +``` + +```yaml +case_id: PIR-VERIFY-06 +title: Recurrence signal reopens a prior incomplete action +current_incident: IR-2026-044 +similar_prior_incidents: + - IR-2026-012 +prior_action: + id: REM-012-03 + status: Implemented + verification: failed +recurrence_signal: + detection: phishing-click-to-oauth-consent + count_30_days: 3 + reopen_threshold: 1 +expected_classification: + status: Action reopened + reason: "A recurring precursor exceeded the threshold and the prior action was never verified." +``` + +```yaml +case_id: PIR-VERIFY-07 +title: Process update requires tabletop evidence +action: + id: REM-007 + mapped_control_failure: process escalation gap + priority: P2 + closure_criterion: responders can execute updated escalation path during tabletop + implementation_artifact: playbook updated + verification_method: tabletop exercise + verification_evidence: missing +expected_classification: + status: Implemented but not verified + reason: "A playbook update needs walkthrough or tabletop evidence before the process action is closed." +```