Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
"skills": [
"./skills/local-ai-use",
"./skills/local-ai-app-integration",
"./skills/rocm-doctor",
"./skills/serving-llms-on-instinct",
"./skills/tracelens-analysis-orchestrator"
],
Expand Down
1 change: 1 addition & 0 deletions .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"skills": [
"./skills/local-ai-use",
"./skills/local-ai-app-integration",
"./skills/rocm-doctor",
"./skills/serving-llms-on-instinct",
"./skills/tracelens-analysis-orchestrator"
],
Expand Down
1 change: 1 addition & 0 deletions .cursor-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"skills": [
"./skills/local-ai-use",
"./skills/local-ai-app-integration",
"./skills/rocm-doctor",
"./skills/serving-llms-on-instinct",
"./skills/tracelens-analysis-orchestrator"
],
Expand Down
53 changes: 0 additions & 53 deletions .github/skillspector-allow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,59 +17,6 @@
# Add entries sparingly and only when the finding is demonstrably benign.

suppressions:
- skill: rocm-doctor
rule: YR1
file: scripts/apply_fix.py
match: backdoor_persistence
reason: >-
False positive. The 'backdoor_persistence' YARA rule's $bashrc_persist
string matches any `echo ... >> ~/.bashrc`. Here it is the documented
remediation that appends `export PATH="/opt/rocm/bin:$PATH"` so ROCm
binaries land on PATH after install. Standard ROCm setup guidance, not a
persistence backdoor or payload.
- skill: rocm-doctor
rule: YR1
file: scripts/diagnose.py
match: backdoor_persistence
reason: >-
False positive. Same $bashrc_persist match: diagnose.py prints the
remediation command `echo 'export PATH=<bin>:$PATH' >> ~/.bashrc` (or
~/.zshrc) for the user to add ROCm/HIP to PATH. No payload, no SSH key
injection, no hidden user.
- skill: rocm-doctor
rule: OH1
file: scripts/apply_fix.py
match: Unvalidated Output Injection
reason: >-
False positive. The flag is on the generic `_run(cmd: list[str], ...)`
helper, which calls `subprocess.run(cmd, ..., shell defaults to False)`
with a list-form argv, so there is no shell interpolation. Every `cmd`
is a hardcoded argv list assembled in-script (e.g.
`["usermod","-a","-G","render,video",user]`, `["modprobe","amdgpu"]`);
the only dynamic pieces are the local username from `$USER`/`$LOGNAME`
and binary paths resolved via `shutil.which`. No LLM/model output ever
reaches this sink, so there is nothing to validate or sanitize.
- skill: rocm-doctor
rule: OH1
file: scripts/examine.py
match: Unvalidated Output Injection
reason: >-
False positive. Same generic `_run(cmd: list[str], ...)` helper as in
apply_fix.py: list-form `subprocess.run` with no shell=True. The read-only
probes only ever pass fixed argv lists (`["rocminfo"]`,
`["lspci","-nn","-D"]`, the PowerShell/CIM `Get-CimInstance` probes, the
framework binary from `shutil.which`). No model output flows into the
command, and there is no shell to inject into.
- skill: rocm-doctor
rule: PE3
file: scripts/examine.py
match: Credential Access
reason: >-
False positive. Line 493 is a code comment ("Resolve uid/gid to names via
/etc/passwd & /etc/group") describing how `_stat_device` maps a device's
owner uid/gid to names. The actual resolution uses the stdlib `pwd`/`grp`
modules (`pwd.getpwuid` / `grp.getgrgid`), not any read of /etc/passwd,
/etc/shadow, .env, or token files. No credential material is accessed.
- skill: local-ai-use
rule: SC2
file: SKILL.md
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Cross-stack skills, from client to cloud.

| Skill | What it does | Source |
| --- | --- | --- |
| `rocm-doctor` | Diagnose ROCm / PyTorch / llama.cpp failures on AMD GPUs against a fixed list of known misconfigurations. | _planned_ |
| [`rocm-doctor`](skills/rocm-doctor/SKILL.md) | Diagnose ROCm / HIP / PyTorch / llama.cpp failures on AMD GPUs (Linux and Windows) against a closed list of known misconfigurations, then fix with consent or route upstream. Thin driver over the `rocm` CLI (`examine` / `diagnose` / `fix`). | in-repo |
| `hyperloom-kernel-optimizer` | Autonomously optimizes LLM inference on AMD GPUs. | _planned_ |
| `vllm-semantic-router` | Setup a vLLM router that semantically maps your request to the best available platform. | _planned_ |
| `hrr-replay-analysis` | Record, replay, and analyze GPU workload behavior on ROCm across AMD Instinct, Radeon, and Ryzen hardware using HIP Record and Replay archives. | _planned_ |
Expand All @@ -102,11 +102,11 @@ A skill is a self-contained folder that bundles everything an agent needs to per

```
skills/
rocm-doctor/
<skill-name>/
SKILL.md
skill-card.md
scripts/
references/
scripts/ # optional
references/ # optional
```

When an agent decides a skill is relevant (or you invoke it explicitly), it loads that `SKILL.md` and follows the instructions inside. Descriptions stay in context cheaply; the full body of a skill only loads when the task actually matches.
Expand Down
139 changes: 139 additions & 0 deletions skills/rocm-doctor/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
name: rocm-doctor
description: >-
Diagnoses why ROCm, the HIP SDK, PyTorch, or llama.cpp is broken on an AMD GPU
on Linux or Windows, then applies a low-risk fix with consent or hands back the
exact next step. Also routes Lemonade, LM Studio, and Ollama problems to the
right upstream channel. Use when the user reports that ROCm or HIP "isn't
working", torch.cuda.is_available() is False, rocminfo / hipInfo can't see the
GPU, or hits hipErrorNoBinaryForGpu, HSA_STATUS_ERROR_INVALID_ISA, "invalid
device function", "no kernel image is available", cannot open /dev/kfd,
permission denied on /dev/kfd, "ROCk module is NOT loaded", a missing
libamdhip64.so / amdhip64_6.dll / hipblas.dll / vcruntime140_1.dll, an
HSA_OVERRIDE_GFX_VERSION page fault, an iGPU+dGPU crash, a container that can't
see the GPU, or an amdgpu-install / DKMS failure. Backed by the `rocm` CLI
(`rocm examine` / `rocm diagnose` / `rocm fix`); this skill is a thin driver
over those commands, not a re-implementation.
---

# ROCm Doctor

Given a "ROCm / PyTorch / llama.cpp isn't working on my AMD GPU" complaint,
identify which **known misconfiguration** is the cause and either fix it (with
consent) or hand back the exact next step.

This skill does **not** probe or reason on its own. The `rocm` CLI owns the
probe, the closed failure-mode catalog, and the fixes; the skill just drives it
and relays the results. The catalog is a **closed list** — if the symptom
doesn't match a known mode, route the user upstream instead of guessing.

## Workflow

0. **Ensure the `rocm` CLI is present.** Everything below shells out to it, so
check first and install it if missing:

```
rocm --version
```

If that succeeds, skip to step 1. If it's not found, install it **with the
user's consent** (this fetches and runs an installer that drops the `rocm` and
`rocmd` binaries into `~/.local/bin`). Only nightly builds are published
today, so install from the `nightly` channel:

- **Linux / macOS:**
```
curl -fsSL https://raw.githubusercontent.com/ROCm/rocm-cli/main/install.sh | sh -s -- nightly
```
- **Windows (PowerShell):**
```
$env:ROCM_CLI_CHANNEL = "nightly"
irm https://raw.githubusercontent.com/ROCm/rocm-cli/main/install.ps1 | iex
```

(Once rocm-cli cuts a stable release, drop the `nightly` channel — `sh` /
`iex` alone will pull the latest stable build.)

After install, confirm `~/.local/bin` is on `PATH` and re-run `rocm --version`.
If it still isn't available, hand the user the install page
(https://github.com/ROCm/rocm-cli) and stop.

1. **Diagnose.** Pass the user's error text as the symptom:

```
rocm diagnose --symptom "<paste the exact error>" --json
```

Read the JSON:
- `matched[]` — ranked causes, each with `id`, `title`, `score` (0–100),
`evidence[]`, and a `fix` (with `fix_id`, `summary`, `commands`, `verify`,
`notes`, and the `needs_sudo` / `needs_reboot` / `needs_relogin` /
`auto_applicable` flags). `score >= 75` = high confidence; `50–74` = likely
(confirm one more piece of evidence with the user first).
- `out_of_scope` — when set (e.g. WSL2), do **not** diagnose. First, if the
user's symptom clearly names an app that ships its own runtime (Lemonade,
Ollama, LM Studio), route them to that app's tracker (see
[Framework routing](#framework-routing)) — those trackers apply regardless
of platform. Otherwise relay the `out_of_scope` message and stop (see
[Out of scope](#out-of-scope)).
- `route_when_no_match` — when `matched` is empty, hand the user this
upstream tracker; **do not speculate**. Note the CLI picks this target from
the *host-detected* framework, not from the symptom text — so for an app
named only in the symptom, route it yourself per
[Framework routing](#framework-routing).

2. **Propose the fix.** Show the top match's `title`, `evidence`, plan, and
`verify` command. Only propose applying it when the user is on board.

3. **Apply with consent.** For an auto-applicable fix:

```
rocm fix <fix-id> # auto fixes: prompt before changing anything
rocm fix <fix-id> --dry-run # show the exact change, touch nothing
rocm fix <fix-id> --yes # required to apply in a non-interactive shell
```

Only the four auto-applicable fixes prompt and mutate. The other 11 are
**print-only** (bootloader, kernel, reinstall, Windows driver, …): `rocm fix
<id>` just prints the plan for the user to run themselves — no prompt, and the
CLI never performs those.

4. **Verify.** Have the user run the `verify` command from the diagnosis.

Use `rocm examine` (or `rocm examine --json`) when you only need the host state
(GPU, driver, ROCm install, groups, framework) without a diagnosis.

## Framework routing

`rocm diagnose` covers frameworks that build against the **system** ROCm/HIP:

- **PyTorch**, **llama.cpp** — in scope; diagnose normally.

Apps that ship their **own** ROCm runtime aren't diagnosed here — route the user
to the right tracker. (The CLI's `route_when_no_match` also targets these, but
only when the host probe *detects* that app; when the app is named only in the
symptom, do the routing yourself using the list below.)

- **Lemonade** → https://github.com/lemonade-sdk/lemonade/issues
- **Ollama** → https://github.com/ollama/ollama/issues
- **LM Studio** → in-app support (no public repo)
- Anything else with no catalog match → ROCm core:
https://github.com/ROCm/ROCm/issues (this is what `route_when_no_match`
returns by default).

## Out of scope

- **WSL2** — a distinct platform (`/dev/dxg` + the Windows host driver, not the
in-tree `amdgpu` module or `/dev/kfd`). `rocm examine`/`diagnose` detect it and
route out; relay that guidance and point at AMD's ROCm-on-WSL guide.
- **NVIDIA / Intel / Apple Silicon GPUs**, and **fresh installs on a clean
machine** (a setup task, not a diagnosis). Exit cleanly and say so.

## Rules

- Never invent a fix. If `rocm diagnose` returns no match, route upstream.
- Never run a mutating fix without the user's explicit OK; prefer `--dry-run`
first. New failure modes are added to the CLI catalog, not improvised here.

See `reference.md` for the full closed catalog and the CLI command/exit-code
reference.
91 changes: 91 additions & 0 deletions skills/rocm-doctor/reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# ROCm Doctor — reference

The closed failure-mode catalog and the CLI it drives. The catalog is
authoritative in the `rocm` CLI (`crates/rocm-core`); this file mirrors it for
humans. To add or change a failure mode, change the CLI catalog — not this doc.

## CLI commands

### `rocm examine [--json]`

Inspect the host: GPU + gfx target, driver, ROCm install, render/video groups,
`/dev/kfd` + render devices, kernel modules, framework introspection, recent
amdgpu kernel-log evidence. `--json` emits the **Examination** document for
tooling.

- It's a general system inspector, so it **always exits 0**. The verdict is the
`status` field: `ok` · `no-amd-gpu` · `wsl` · `unsupported-os` · `degraded`.

### `rocm diagnose [--symptom "<text>"] [--top N] [--json]`

Match the host + symptom against the closed catalog. **Always exits 0**; read
the result from `--json`:

- `matched[]` — ranked `{ id, title, score, evidence[], fix }`. Tiers:
`>= 75` high confidence, `50–74` likely, `< 50` weak.
- `min_score_for_match` (50), `high_confidence_threshold` (75).
- `out_of_scope` — set when the host is off-catalog (e.g. WSL2); `matched` is empty.
- `route_when_no_match` — `{ target, url }` upstream tracker to use when nothing matched.

### `rocm fix [<id>] [--yes] [--dry-run] [--device-index N]`

Apply a fix by id (run with no id to list). Exit codes:

| code | meaning |
| --- | --- |
| 0 | applied / dry-run / print-only plan / list |
| 1 | internal error |
| 2 | usage error (incl. unknown fix-id) |
| 3 | not applicable on this host (OS mismatch, missing/negative `--device-index`) — nothing changed |
| 4 | attempted but the command failed |
| 5 | user declined at the prompt |

Only four fixes are auto-applicable — `fix-2-unset-override`,
`fix-4-render-group`, `fix-6-path`, `fix-9-igpu-dgpu` — and the rest print their
plan for the user to run. Pass the **full** id (`rocm fix fix-2-unset-override`,
not `rocm fix fix-2`; a short id returns exit 2, unknown fix-id). Auto fixes
print the exact command, honor `--dry-run`, refuse on a non-interactive shell
without `--yes`, and confirm before mutating.

## Closed catalog (15 failure modes)

| id | OS | Failure mode | Typical signal | Auto-fix |
| --- | --- | --- | --- | --- |
| `fix-1-arch` | both | GPU gfx target not in the framework's build arch list | `hipErrorNoBinaryForGpu`, `HSA_STATUS_ERROR_INVALID_ISA`, "invalid device function" | no |
| `fix-2-unset-override` | both | `HSA_OVERRIDE_GFX_VERSION` set on a GPU that now has a native wheel | page faults / `OUT_OF_REGISTERS`, override set in env | yes |
| `fix-3-rocm-kernel` | linux | ROCm + distro/kernel form an unsupported triple | ROCm installed but `amdgpu` not loaded; DKMS build failure | no |
| `fix-4-render-group` | linux | User not in `render`/`video` group (or `/dev/kfd` owned by the other group) | cannot open `/dev/kfd`, permission denied | yes |
| `fix-5-amdgpu-load` | linux | `amdgpu` module not loaded (or blacklisted) | "ROCk module is NOT loaded", blacklist entry, Secure Boot | no |
| `fix-6-path` | both | ROCm/HIP binaries not on PATH after install | `rocminfo: command not found`, `hipInfo` missing from PATH | yes |
| `fix-7-stale-repos` | linux | Stale/conflicting APT/DNF repos from prior installer runs | apt 404 `repo.radeon.com`, unmet deps, ≥2 ROCm repo files | no |
| `fix-8-wheel-rocm` | both | Framework wheel built for a different ROCm major than the system | `libamdhip64.so.X` / `amdhip64_X.dll` load failure | no |
| `fix-9-igpu-dgpu` | both | iGPU enumerated alongside dGPU, destabilising the runtime | APU + discrete AMD present, `HIP_VISIBLE_DEVICES` unset, crash/segfault | yes |
| `fix-10-container` | linux | Container can't see `/dev/kfd` or `/dev/dri/renderD*` | running in docker/podman, kfd/render devices missing | no |
| `fix-11-iommu` | linux | Multi-GPU hang with IOMMU enabled | ≥2 AMD GPUs, `iommu=` not `pt`, hang/deadlock/timeout | no |
| `fix-12-installer` | linux | `amdgpu-install` left a broken DKMS / repo state | dpkg half-configured, DKMS failed, `--accept-eula` | no |
| `fix-13-hip-sdk-missing` | windows | HIP SDK not installed | no HIP SDK under Program Files, `hipInfo` not recognized | no |
| `fix-14-adrenalin-too-old` | windows | Adrenalin / kernel-mode driver too old for the HIP SDK | `hipInfo` can't enumerate, "driver too old", HSA "no agents found" | no |
| `fix-15-msvc-redist` | windows | MSVC runtime missing (HIP DLLs can't load) | `vcruntime140.dll` / `vcruntime140_1.dll` missing | no |

Linux-only: fix-3, -4, -5, -7, -10, -11, -12. Windows-only: fix-13, -14, -15.
Cross-platform: fix-1, -2, -6, -8, -9.

## Framework routing

`rocm diagnose` diagnoses frameworks built against the **system** ROCm/HIP:

- **PyTorch**, **llama.cpp** — in scope.

Apps that ship their own runtime are routed upstream (via `route_when_no_match`):

- **Lemonade** → https://github.com/lemonade-sdk/lemonade/issues
- **Ollama** → https://github.com/ollama/ollama/issues
- **LM Studio** → in-app support (no public repo)
- Otherwise → ROCm core: https://github.com/ROCm/ROCm/issues

## Out of scope

- **WSL2** — distinct platform (`/dev/dxg` + Windows host driver). Detected and
routed out; point at AMD's ROCm-on-WSL guide:
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installryz/wsl/howto_wsl.html
- NVIDIA / Intel / Apple Silicon GPUs; fresh installs on a clean machine.
17 changes: 17 additions & 0 deletions skills/rocm-doctor/skill-card.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Skill Card

## Description

Diagnoses ROCm / HIP SDK / PyTorch / llama.cpp failures on AMD GPUs (Linux and
Windows) against a closed list of known misconfigurations, and applies a
low-risk fix with consent or routes the user to the right upstream channel. A
thin driver over the `rocm` CLI (`rocm examine` / `rocm diagnose` / `rocm fix`) —
the probe, catalog, and fixes live in the CLI, versioned with the binary.

## Owner

rocm-cli team (AMD)

## License

MIT
Loading
Loading