-
Notifications
You must be signed in to change notification settings - Fork 14
OCPEDGE-2727: Add eval harness configs for cluster-diagnostic and threat-model skills #189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
25b476c
Add threat-model skill plugin with TNF, TNA, SNO, and LVMS support
dhensel-rh bbfce72
fix: add missing blank lines to pass markdownlint (MD031/MD032)
dhensel-rh a74506d
Shorten threat-model skill directory names and extract shared report …
dhensel-rh 8477584
Support REPORT_DIR env var override in threat-model skills
dhensel-rh 4ede53f
OCPEDGE-2727: Add eval harness configs for cluster-diagnostic and thr…
dhensel-rh dd1d402
Fix dataset.path to use repo-root-relative paths
dhensel-rh c63b86f
Update evals README with detailed pipeline steps
dhensel-rh 11404a4
Add game mode eval case and improve judges
dhensel-rh d5874c4
Set REPORT_DIR in threat-model eval and improve README framing
dhensel-rh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../plugins/threat-model/skills/lvms-threat-model |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../plugins/threat-model/skills/sno-threat-model |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../plugins/threat-model/skills/tna-threat-model |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../plugins/threat-model/skills/tnf-threat-model |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "name": "threat-model", | ||
| "version": "1.0.0", | ||
| "description": "Analyze PRs for security threats with STRIDE/DFD analysis, MITRE ATT&CK and OWASP mapping for OpenShift edge topologies (TNA, TNF, SNO, LVMS)", | ||
| "author": { | ||
| "name": "TNF Security" | ||
| }, | ||
| "license": "Apache-2.0", | ||
| "keywords": ["security", "threat-model", "stride", "mitre", "owasp", "openshift", "tnf", "tna", "sno", "lvms"] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Threat Model Plugin for Claude Code | ||
|
|
||
| Security threat analysis for OpenShift PRs across multiple topologies (TNF, TNA, SNO, LVMS). | ||
|
|
||
| ## What It Does | ||
|
|
||
| Analyzes pull requests for security threats against OpenShift clusters: | ||
|
|
||
| - Fetches PR diffs from GitHub | ||
| - Runs ShellCheck on shell scripts | ||
| - Maps changes to Data Flow Diagram (DFD) elements | ||
| - Applies per-element STRIDE analysis | ||
| - Cross-references against formal threat models | ||
| - Maps findings to MITRE ATT&CK techniques and OWASP Top 10:2025 | ||
| - Generates formal threat analysis reports | ||
|
|
||
| ## Usage | ||
|
|
||
| ### TNF (Two-Node Fencing) | ||
|
|
||
| ```bash | ||
| /threat-model:tnf 2136 | ||
| /threat-model:tnf https://github.com/ClusterLabs/resource-agents/pull/2136 | ||
| /threat-model:tnf resource-agents 2136 | ||
| ``` | ||
|
|
||
| ### TNA (Two-Node Arbiter) | ||
|
|
||
| ```bash | ||
| /threat-model:tna 1437 | ||
| /threat-model:tna https://github.com/openshift/cluster-etcd-operator/pull/1437 | ||
| /threat-model:tna installer 10403 | ||
| ``` | ||
|
|
||
| ### SNO (Single Node OpenShift) | ||
|
|
||
| ```bash | ||
| /threat-model:sno 10498 | ||
| /threat-model:sno https://github.com/openshift/installer/pull/10498 | ||
| /threat-model:sno installer 10498 | ||
| ``` | ||
|
|
||
| ### LVMS (LVM Storage) | ||
|
|
||
| ```bash | ||
| /threat-model:lvms 2271 | ||
| /threat-model:lvms https://github.com/openshift/lvm-operator/pull/2271 | ||
| /threat-model:lvms lvm-operator 2271 | ||
| ``` | ||
|
|
||
| > **Note**: The LVMS DFD model is not yet defined. The LVMS skill performs general security analysis, ShellCheck scanning, and MITRE/OWASP mapping. Full DFD/STRIDE analysis will be available once its DFD model is created. | ||
|
|
||
| ## Workspace Requirements | ||
|
|
||
| The skill expects a workspace with a `repos/` directory containing cloned repositories. It auto-discovers the workspace root at runtime. | ||
|
|
||
| ### Recommended workspace layout | ||
|
|
||
| ```text | ||
| your-workspace/ | ||
| ├── repos/ | ||
| │ ├── cluster-etcd-operator/ | ||
| │ ├── installer/ | ||
| │ ├── machine-config-operator/ | ||
| │ ├── resource-agents/ | ||
| │ ├── two-node-toolbox/ | ||
| │ │ └── docs/ | ||
| │ │ ├── TNF-THREAT-MODEL.md | ||
| │ │ └── TNA-THREAT-MODEL.md | ||
| │ └── ... | ||
| └── .claude/ | ||
| └── skills/ | ||
| ├── threat-model/ | ||
| ├── mitre-findings-tnf.md # Created automatically on first use | ||
| ├── mitre-findings-tna.md | ||
| ├── mitre-findings-sno.md | ||
| └── mitre-findings-lvms.md | ||
| ``` | ||
|
|
||
| ### Optional dependencies | ||
|
|
||
| - **ShellCheck** (`dnf install ShellCheck`) - for automated shell script analysis | ||
| - **gh** CLI - for fetching PR details from GitHub | ||
| - **Formal threat model files** - for DFD/STRIDE cross-referencing | ||
|
|
||
| ## What's Included | ||
|
|
||
| | File | Purpose | | ||
| |------|---------| | ||
| | `skills/tnf-threat-model/SKILL.md` | TNF threat analysis skill | | ||
| | `skills/tnf-threat-model/dfd-elements-tnf.md` | TNF DFD element catalog | | ||
| | `skills/tna-threat-model/SKILL.md` | TNA threat analysis skill | | ||
| | `skills/tna-threat-model/dfd-elements-tna.md` | TNA DFD element catalog | | ||
| | `skills/sno-threat-model/SKILL.md` | SNO threat analysis skill | | ||
| | `skills/sno-threat-model/dfd-elements-sno.md` | SNO DFD element catalog (SNO-P1–P6, SNO-DS1–DS6, SNO-DF1–DF10, SNO-TB1–TB3) | | ||
| | `skills/lvms-threat-model/SKILL.md` | LVMS threat analysis skill | | ||
| | `skills/lvms-threat-model/dfd-elements-lvms.md` | LVMS DFD element catalog (placeholder) | | ||
| | `references/mitre-reference.md` | MITRE ATT&CK quick reference | | ||
| | `references/owasp-reference.md` | OWASP Top 10:2025 reference | | ||
| | `references/mitre-findings-template.md` | Cumulative findings tracker template | | ||
|
|
||
| ## License | ||
|
|
||
| Apache-2.0 | ||
13 changes: 13 additions & 0 deletions
13
plugins/threat-model/references/mitre-findings-template.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # MITRE ATT&CK Findings Tracker | ||
|
|
||
| Cumulative security findings from PR threat analysis. | ||
|
|
||
| ## Legend | ||
|
|
||
| **Severity**: Critical / High / Medium / Low / Info | ||
| **Status**: Open / Mitigated / Accepted / FalsePositive | ||
|
|
||
| --- | ||
|
|
||
| <!-- APPEND-ONLY below this line. Each skill writes to its own file (mitre-findings-<topology>.md). --> | ||
| <!-- See SKILL.md § "Findings File" for the append protocol. --> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| # MITRE ATT&CK Quick Reference | ||
|
|
||
| Common techniques for infrastructure and Kubernetes security. | ||
|
|
||
| ## Initial Access (TA0001) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1078 | Valid Accounts | Default creds, leaked tokens, service account abuse | | ||
| | T1190 | Exploit Public-Facing App | Unpatched CVEs, injection flaws | | ||
| | T1133 | External Remote Services | Exposed SSH, RDP, VNC, API endpoints | | ||
|
|
||
| ## Execution (TA0002) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1059 | Command/Script Interpreter | Shell exec, eval, unsanitized input to commands | | ||
| | T1609 | Container Admin Command | kubectl exec, docker exec, crictl | | ||
| | T1610 | Deploy Container | Malicious container images, privileged pods | | ||
|
|
||
| ## Persistence (TA0003) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1053 | Scheduled Task/Job | CronJobs, systemd timers | | ||
| | T1098 | Account Manipulation | Adding users, modifying RBAC | | ||
| | T1543 | Create/Modify System Process | Systemd services, init scripts | | ||
| | T1136 | Create Account | New ServiceAccounts, local users | | ||
|
|
||
| ## Privilege Escalation (TA0004) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1068 | Exploitation for Privilege Escalation | CVE exploits, kernel vulns | | ||
| | T1548 | Abuse Elevation Control | sudo, setuid, capabilities | | ||
| | T1611 | Escape to Host | Container breakout, hostPID, hostNetwork | | ||
|
|
||
| ## Defense Evasion (TA0005) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1070 | Indicator Removal | Log deletion, history clearing | | ||
| | T1562 | Impair Defenses | Disabling SELinux, seccomp, audit | | ||
| | T1036 | Masquerading | Renamed binaries, fake processes | | ||
|
|
||
| ## Credential Access (TA0006) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1552 | Unsecured Credentials | Hardcoded secrets, env vars, config files | | ||
| | T1528 | Steal Application Access Token | Token theft, SA token access | | ||
| | T1003 | OS Credential Dumping | /etc/shadow, memory scraping | | ||
| | T1555 | Credentials from Password Stores | Secret managers, keyrings | | ||
|
|
||
| ## Discovery (TA0007) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1083 | File and Directory Discovery | Filesystem enumeration | | ||
| | T1046 | Network Service Discovery | Port scanning, service probing | | ||
| | T1613 | Container and Resource Discovery | kubectl get, API enumeration | | ||
|
|
||
| ## Lateral Movement (TA0008) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1021 | Remote Services | SSH, WinRM, kubectl | | ||
| | T1550 | Use Alternate Auth Material | Token reuse, cert theft | | ||
|
|
||
| ## Impact (TA0040) | ||
|
|
||
| | ID | Technique | Indicators | | ||
| |----|-----------|------------| | ||
| | T1485 | Data Destruction | rm -rf, etcd data deletion | | ||
| | T1486 | Data Encrypted for Impact | Ransomware patterns | | ||
| | T1489 | Service Stop | systemctl stop, kill processes | | ||
| | T1529 | System Shutdown/Reboot | STONITH abuse, power off | | ||
|
|
||
| ## TNF-Specific Techniques | ||
|
|
||
| | ID | Technique | TNF Context | DFD Elements | | ||
| |----|-----------|-------------|--------------| | ||
| | T1552 | Unsecured Credentials | BMC credentials in install-config, secrets, CIB | DS1, DS2, DS3, DF1-DF9 | | ||
| | T1529 | System Shutdown | Malicious fencing, STONITH abuse | P6, P8, EE2 | | ||
| | T1489 | Service Stop | etcd/pacemaker service disruption | P7, DS5 | | ||
| | T1557 | Adversary-in-the-Middle | BMC MITM when cert disabled, Corosync interception | P8, DF10, EE2, EE3 | | ||
| | T1078 | Valid Accounts | BMC account compromise, predictable PCSD token | P3, P8, DS4, EE2 | | ||
| | T1059 | Command Interpreter | Shell injection via credentials, OCF agent scripts | P5, P7, DF9 | | ||
| | T1611 | Escape to Host | Privileged TNF setup/fencing containers with nsenter | P3, P4, P5 | | ||
| | T1562 | Impair Defenses | CIB manipulation to disable STONITH | DS3, P4 | | ||
|
|
||
| ## TNA-Specific Techniques | ||
|
|
||
| | ID | Technique | TNA Context | DFD Elements | | ||
| |----|-----------|-------------|--------------| | ||
| | T1078 | Valid Accounts | Admin credential theft (kubeconfig) | TNA-EE1 | | ||
| | T1552 | Unsecured Credentials | Worker ignition token leak | TNA-DS6 | | ||
| | T1611 | Escape to Host | Container escape from pod to node root | TNA-P5 | | ||
| | T1562 | Impair Defenses | Arbiter taint removal disabling scheduling protection | TNA-P3 | | ||
| | T1489 | Service Stop | etcd quorum disruption (arbiter + 1 master) | TNA-DS5 | | ||
| | T1021 | Remote Services | Lateral movement from worker to control plane via pod network | TNA-P5, TNA-DS5 | | ||
|
|
||
| ## SNO-Specific Techniques | ||
|
|
||
| | ID | Technique | SNO Context | DFD Elements | | ||
| |----|-----------|-------------|--------------| | ||
| | T1552 | Unsecured Credentials | Pull secret, offline token, kubeadmin-password on admin workstation | SNO-DS1, SNO-DS4 | | ||
| | T1611 | Escape to Host | Bootstrap-in-place agent runs privileged on bare metal | SNO-P5 | | ||
| | T1610 | Deploy Container | Workloads scheduled on master (no worker isolation) | SNO-P6 | | ||
| | T1562 | Impair Defenses | UnsafeScalingStrategy bypasses quorum safety checks | SNO-P4 | | ||
| | T1485 | Data Destruction | Single etcd member — node failure = total data loss | SNO-DS3 | | ||
| | T1195 | Supply Chain Compromise | Discovery ISO tampering before boot | SNO-DS2 | | ||
| | T1078 | Valid Accounts | Admin credential theft (kubeconfig) | SNO-EE1 | | ||
|
|
||
| ## LVMS-Specific Techniques | ||
|
|
||
| > **Not yet defined.** This section will be populated once the LVMS DFD model is created. | ||
|
|
||
| ## TNF DFD Element to ATT&CK Mapping | ||
|
|
||
| | DFD Element | Primary ATT&CK Techniques | Per-Element STRIDE IDs | | ||
| |-------------|--------------------------|----------------------| | ||
| | P1 (Installer) | T1552 | PE-P1-I-1, PE-P1-T-1 | | ||
| | P3 (Auth Job) | T1078, T1611 | PE-P3-S-1, PE-P3-E-1 | | ||
| | P4 (Setup Job) | T1611, T1562 | PE-P4-E-1, PE-P4-T-1 | | ||
| | P5 (Fencing Job) | T1059, T1552, T1611 | PE-P5-I-1, PE-P5-T-1, PE-P5-E-1 | | ||
| | P6 (fenced) | T1529 | PE-P6-S-1, PE-P6-D-1 | | ||
| | P7 (podman-etcd) | T1489, T1059 | PE-P7-T-1, PE-P7-D-1 | | ||
| | P8 (fence_redfish) | T1557, T1529, T1552 | PE-P8-S-1, PE-P8-I-1 | | ||
| | DS1 (install-config) | T1552 | PE-DS1-I-1 | | ||
| | DS2 (K8s Secrets) | T1552 | PE-DS2-I-1, PE-DS2-T-1 | | ||
| | DS3 (CIB) | T1552, T1562 | PE-DS3-I-1, PE-DS3-T-1 | | ||
| | DS4 (PCSD Token) | T1078 | PE-DS4-I-1 | | ||
| | DF9 (creds as CLI args) | T1552, T1059 | PE-DF9-I-1 | | ||
| | DF10 (Redfish HTTPS) | T1557 | PE-DF10-T-1, PE-DF10-I-1 | | ||
| | EE2 (BMC) | T1529, T1190 | PE-EE2-S-1, PE-EE2-S-2 | | ||
|
|
||
| ## TNA DFD Element to ATT&CK Mapping | ||
|
|
||
| | DFD Element | Primary ATT&CK Techniques | Per-Element STRIDE IDs | | ||
| |-------------|--------------------------|----------------------| | ||
| | TNA-P1 (Installer) | T1552 | PE-TNA-P1-T-1, PE-TNA-P1-D-1 | | ||
| | TNA-P3 (MCO) | T1562 | PE-TNA-P3-T-1, PE-TNA-P3-D-1 | | ||
| | TNA-P4 (CEO) | T1489 | PE-TNA-P4-T-1, PE-TNA-P4-D-1 | | ||
| | TNA-P5 (Worker Kubelet) | T1021, T1611 | PE-TNA-P5-S-1, PE-TNA-P5-E-1 | | ||
| | TNA-DS5 (etcd Data) | T1552, T1489 | PE-TNA-DS5-T-1, PE-TNA-DS5-I-1, PE-TNA-DS5-D-1 | | ||
| | TNA-DS6 (Worker Ignition) | T1552 | PE-TNA-DS6-T-1, PE-TNA-DS6-I-1 | | ||
| | TNA-EE1 (Admin) | T1078 | PE-TNA-EE1-S-1, PE-TNA-EE1-R-1 | | ||
|
|
||
| ## SNO DFD Element to ATT&CK Mapping | ||
|
|
||
| | DFD Element | Primary ATT&CK Techniques | Per-Element STRIDE IDs | | ||
| |-------------|--------------------------|----------------------| | ||
| | SNO-P1 (Installer) | T1552 | PE-SNO-P1-T-1, PE-SNO-P1-I-1 | | ||
| | SNO-P2 (Assisted Service) | T1078, T1552 | PE-SNO-P2-S-1, PE-SNO-P2-T-1 | | ||
| | SNO-P3 (MCO) | T1611 | PE-SNO-P3-T-1, PE-SNO-P3-E-1 | | ||
| | SNO-P4 (CEO) | T1562 | PE-SNO-P4-T-1, PE-SNO-P4-D-1 | | ||
| | SNO-P5 (Bootstrap Agent) | T1611, T1552 | PE-SNO-P5-E-1, PE-SNO-P5-I-1 | | ||
| | SNO-P6 (Kubelet) | T1610 | PE-SNO-P6-E-1, PE-SNO-P6-D-1 | | ||
| | SNO-DS1 (install-config) | T1552 | PE-SNO-DS1-I-1 | | ||
| | SNO-DS2 (Discovery ISO) | T1195 | PE-SNO-DS2-T-1, PE-SNO-DS2-I-1 | | ||
| | SNO-DS3 (etcd Data) | T1485, T1552 | PE-SNO-DS3-T-1, PE-SNO-DS3-I-1, PE-SNO-DS3-D-1 | | ||
| | SNO-DS4 (Credentials) | T1552 | PE-SNO-DS4-I-1 | | ||
| | SNO-EE1 (Admin) | T1078 | PE-SNO-EE1-S-1, PE-SNO-EE1-R-1 | | ||
| | SNO-EE2 (Assisted Service Cloud) | T1195 | PE-SNO-EE2-S-1 | | ||
|
|
||
| ## References | ||
|
|
||
| - MITRE ATT&CK Enterprise: <https://attack.mitre.org/matrices/enterprise/> | ||
| - MITRE ATT&CK Containers: <https://attack.mitre.org/matrices/enterprise/containers/> | ||
| - MITRE ATT&CK Mitigations: <https://attack.mitre.org/mitigations/enterprise/> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Show the actual
.claude/skillslayout here.The diagram currently suggests a
.claude/skills/threat-model/subdirectory, but the plugin is installed as four top-level symlinks directly under.claude/skills/. As written, this example points users at the wrong path.🔧 Suggested fix
└── .claude/ └── skills/ - ├── threat-model/ + ├── threat-model-tnf + ├── threat-model-tna + ├── threat-model-sno + ├── threat-model-lvms ├── mitre-findings-tnf.md # Created automatically on first use ├── mitre-findings-tna.md ├── mitre-findings-sno.md └── mitre-findings-lvms.md🤖 Prompt for AI Agents