From 03888ebad3d6441a857617c3bb6e9f17a1aa7e44 Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Mon, 15 Jun 2026 09:20:16 +0200 Subject: [PATCH 1/7] feat: add cluster-troubleshoot skill Add OpenShift cluster diagnostics skill for investigating live cluster issues (pod crashes, node failures, operator degradation, etc.) using oc commands and Prometheus metrics. Signed-off-by: Alberto Falossi Assisted-by: Claude Code:claude-opus-4-6 --- cluster-troubleshoot/OWNERS | 16 ++++++ cluster-troubleshoot/README.md | 3 ++ cluster-troubleshoot/SKILL.md | 90 ++++++++++++++++++++++++++++++++++ 3 files changed, 109 insertions(+) create mode 100644 cluster-troubleshoot/OWNERS create mode 100644 cluster-troubleshoot/README.md create mode 100644 cluster-troubleshoot/SKILL.md diff --git a/cluster-troubleshoot/OWNERS b/cluster-troubleshoot/OWNERS new file mode 100644 index 0000000..695ea2e --- /dev/null +++ b/cluster-troubleshoot/OWNERS @@ -0,0 +1,16 @@ +# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md + +approvers: + - falox + - tremes + - iNecas + - harche + - mrunalp +reviewers: + - falox + - tremes + - iNecas + - harche + - mrunalp + +component: "Cluster Troubleshooting Lightspeed Skills" diff --git a/cluster-troubleshoot/README.md b/cluster-troubleshoot/README.md new file mode 100644 index 0000000..c5d37c7 --- /dev/null +++ b/cluster-troubleshoot/README.md @@ -0,0 +1,3 @@ +# Cluster Troubleshooting skills + +This directory contains skills which are designed to help agents diagnose and troubleshoot OpenShift cluster issues. diff --git a/cluster-troubleshoot/SKILL.md b/cluster-troubleshoot/SKILL.md new file mode 100644 index 0000000..d42b228 --- /dev/null +++ b/cluster-troubleshoot/SKILL.md @@ -0,0 +1,90 @@ +--- +name: cluster-troubleshoot +description: Diagnose and troubleshoot OpenShift cluster issues. Use when the user reports alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, HTTP errors, DNS failures, or any cluster anomaly. Not for cluster setup, configuration how-tos, or writing alerting rules. +--- + +# ROLE + +You are an expert OpenShift site-reliability engineer. You investigate cluster problems methodically: gather evidence first, then diagnose. + +# ENVIRONMENT + +- Platform: OpenShift Container Platform (OCP). Use OpenShift-specific resources (ClusterOperator, ClusterVersion, MachineConfigPool, Route, etc.) alongside standard Kubernetes ones. +- You run `oc` commands against the live cluster. + +# RESPONSE RULES + +- Verify every claim with evidence from command output. Provide exact resource names, namespaces, timestamps, and error messages. +- If multiple causes exist, list them numbered with supporting evidence. +- If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information. +- In diagnosis mode, stay focused on the reported issue — don't surface unrelated errors. +- No URLs unless from command output or provided context. + +# INVESTIGATION PROTOCOL + +When diagnosing a specific symptom: + +1. **Scope the blast radius** — is it one pod, one node, one namespace, or cluster-wide? This determines which layer to start from. + +2. **Gather evidence in parallel** — run independent `oc` commands together. Inspect the owner workload, pods, logs, events, services, and routes. Check pod conditions and container statuses, not just the phase — a pod can show "Running" but have failing health probes. + +3. **Trace causality chains** — if resource A fails because of B, investigate B. Common chains in OpenShift: + - Pod pending → node pressure, unschedulable nodes, resource quota, or PVC not bound + - Pod crash-looping → OOMKilled (check limits vs. actual usage), application error (check logs), misconfigured probes + - Service unreachable → no endpoints → pods not ready → failing readiness probe + - Operator degraded → operand pod failing → node issue or config error + - Node NotReady → kubelet issues → disk pressure, certificate expired, MCO update stuck, or kernel panic + +4. **Check recent changes** — many issues are caused by something that just changed. Compare symptom onset with rollout history, image tag changes, config/secret edits, HPA scaling, operator upgrades, node drains, and MachineConfig updates. Use `oc rollout history`, `oc describe`, and events sorted by time. + +5. **Keep digging** — after identifying a root cause, continue investigating to collect exact names, versions, and labels, and to check for additional contributing factors. + +6. **Recommend a fix** if you know one, with the exact commands to run. Otherwise suggest mitigations and note whether each is reversible. + +# TOOL USAGE + +- List actual resources before inspecting them — never guess pod names, use label selectors or owner references. +- Sample up to 3 representative pods per workload, not all. +- Use `-o json | jq` for structured data extraction. +- Do not repeat the same command with the same arguments. +- Do not ask the user to run a command — gather the information yourself. + +# QUERYING PROMETHEUS + +OpenShift exposes Prometheus metrics through the Thanos Querier in the `openshift-monitoring` namespace. Query it via the external route, authenticating with your token. + +**Setup** — run once per session: +```bash +TOKEN=$(oc whoami -t) +THANOS_URL=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}') +``` + +**Instant query** (current value): +```bash +wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ + "https://${THANOS_URL}/api/v1/query?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')" | jq . +``` + +**Range query** (time series, e.g. last hour with 60s resolution): +```bash +wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ + "https://${THANOS_URL}/api/v1/query_range?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')&start=$(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ)&end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&step=60s" | jq . +``` + +**Discover available metrics** (never guess metric names): +```bash +wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ + "https://${THANOS_URL}/api/v1/label/__name__/values" | jq '.data[]' | grep -i '' +``` + +**Get firing alerts:** +```bash +wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ + "https://${THANOS_URL}/api/v1/alerts" | jq '.data.alerts[] | select(.state=="firing")' +``` + +**Workflow:** Start by checking firing alerts — their labels contain exact identifiers (namespace, pod, node) that make follow-up queries precise. Always discover metric names before querying. Use instant queries for current state, range queries for trends. If a metric doesn't exist, tell the user — do not fabricate PromQL. + +# STYLE + +- Be highly concise. Evidence-backed conclusions, no filler. From 5157f20aefb32ce01dfb0df96eb51f2a6092583b Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Thu, 9 Jul 2026 16:05:26 +0200 Subject: [PATCH 2/7] chore: update cluster-troubleshoot OWNERS Signed-off-by: Alberto Falossi --- cluster-troubleshoot/OWNERS | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cluster-troubleshoot/OWNERS b/cluster-troubleshoot/OWNERS index 695ea2e..56f9e98 100644 --- a/cluster-troubleshoot/OWNERS +++ b/cluster-troubleshoot/OWNERS @@ -3,14 +3,14 @@ approvers: - falox - tremes + - rioloc + - shwetaap - iNecas - - harche - - mrunalp reviewers: - falox - tremes + - rioloc + - shwetaap - iNecas - - harche - - mrunalp component: "Cluster Troubleshooting Lightspeed Skills" From 817eb46cc774d07ab8bf33405d88641981e4680c Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Thu, 9 Jul 2026 16:06:55 +0200 Subject: [PATCH 3/7] fix: improve cluster-troubleshoot skill instructions Remove ROLE section, add available CLI tools and writable directories to ENVIRONMENT, restrict remediation output to the specific alert being analyzed, and drop curly braces from THANOS_URL references to avoid ADK template substitution errors. Signed-off-by: Alberto Falossi Assisted-by: Claude Code:claude-opus-4-6 --- cluster-troubleshoot/SKILL.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/cluster-troubleshoot/SKILL.md b/cluster-troubleshoot/SKILL.md index d42b228..dc65030 100644 --- a/cluster-troubleshoot/SKILL.md +++ b/cluster-troubleshoot/SKILL.md @@ -3,14 +3,16 @@ name: cluster-troubleshoot description: Diagnose and troubleshoot OpenShift cluster issues. Use when the user reports alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, HTTP errors, DNS failures, or any cluster anomaly. Not for cluster setup, configuration how-tos, or writing alerting rules. --- -# ROLE - -You are an expert OpenShift site-reliability engineer. You investigate cluster problems methodically: gather evidence first, then diagnose. - # ENVIRONMENT - Platform: OpenShift Container Platform (OCP). Use OpenShift-specific resources (ClusterOperator, ClusterVersion, MachineConfigPool, Route, etc.) alongside standard Kubernetes ones. -- You run `oc` commands against the live cluster. +- You run commands against the live cluster. +- Available CLIs: oc, kubectl, jq, wget, openssl, skopeo, python3 +- Networking diagnostics: iproute (ip, ss), bind-utils (dig, nslookup, host), net-tools (netstat, ifconfig), tcpdump, lsof +- Process and system: procps-ng (ps, top, free), strace, findutils, file, diffutils +- Container images: skopeo +- General: git, less, vim, tar, gzip, unzip, diff +- Writable directories: /home/agent, /tmp/agent-workspace. The root filesystem is read-only. # RESPONSE RULES @@ -18,6 +20,7 @@ You are an expert OpenShift site-reliability engineer. You investigate cluster p - If multiple causes exist, list them numbered with supporting evidence. - If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information. - In diagnosis mode, stay focused on the reported issue — don't surface unrelated errors. +- CRITICAL: The output remediation plan and options MUST address only the root cause of the specific alert or issue being analyzed. Never include secondary issues, unrelated findings, or general recommendations discovered during investigation. - No URLs unless from command output or provided context. # INVESTIGATION PROTOCOL @@ -62,25 +65,25 @@ THANOS_URL=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{. **Instant query** (current value): ```bash wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://${THANOS_URL}/api/v1/query?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')" | jq . + "https://$THANOS_URL/api/v1/query?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')" | jq . ``` **Range query** (time series, e.g. last hour with 60s resolution): ```bash wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://${THANOS_URL}/api/v1/query_range?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')&start=$(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ)&end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&step=60s" | jq . + "https://$THANOS_URL/api/v1/query_range?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')&start=$(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ)&end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&step=60s" | jq . ``` **Discover available metrics** (never guess metric names): ```bash wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://${THANOS_URL}/api/v1/label/__name__/values" | jq '.data[]' | grep -i '' + "https://$THANOS_URL/api/v1/label/__name__/values" | jq '.data[]' | grep -i '' ``` **Get firing alerts:** ```bash wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://${THANOS_URL}/api/v1/alerts" | jq '.data.alerts[] | select(.state=="firing")' + "https://$THANOS_URL/api/v1/alerts" | jq '.data.alerts[] | select(.state=="firing")' ``` **Workflow:** Start by checking firing alerts — their labels contain exact identifiers (namespace, pod, node) that make follow-up queries precise. Always discover metric names before querying. Use instant queries for current state, range queries for trends. If a metric doesn't exist, tell the user — do not fabricate PromQL. From a1a88456f0a2ff19f69da784cc357d4df359937e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1=C5=A1=20Reme=C5=A1?= Date: Fri, 17 Jul 2026 08:30:19 +0200 Subject: [PATCH 4/7] feat: add diagnostic tools and evals for cluster-troubleshoot --- cluster-troubleshoot/README.md | 32 +++- cluster-troubleshoot/SKILL.md | 148 +++++++++++----- .../tools/check-recent-changes.sh | 86 +++++++++ cluster-troubleshoot/tools/common.sh | 125 +++++++++++++ cluster-troubleshoot/tools/diagnose-node.sh | 57 ++++++ .../tools/diagnose-operator.sh | 51 ++++++ cluster-troubleshoot/tools/diagnose-pod.sh | 102 +++++++++++ .../tools/diagnose-workload.sh | 102 +++++++++++ .../tools/discover-metrics.sh | 23 +++ .../tools/fetch-alert-rule.sh | 31 ++++ cluster-troubleshoot/tools/get-events.sh | 53 ++++++ .../tools/get-firing-alerts.sh | 25 +++ .../tools/prometheus-query-range.sh | 39 ++++ .../tools/prometheus-query.sh | 14 ++ .../tools/prometheus-setup.sh | 27 +++ cluster-troubleshoot/tools/query-alert.sh | 25 +++ .../cluster-troubleshoot/system_prompt.md | 18 ++ .../cluster-troubleshoot/test_cases.yaml | 167 ++++++++++++++++++ 18 files changed, 1075 insertions(+), 50 deletions(-) create mode 100755 cluster-troubleshoot/tools/check-recent-changes.sh create mode 100755 cluster-troubleshoot/tools/common.sh create mode 100755 cluster-troubleshoot/tools/diagnose-node.sh create mode 100755 cluster-troubleshoot/tools/diagnose-operator.sh create mode 100755 cluster-troubleshoot/tools/diagnose-pod.sh create mode 100755 cluster-troubleshoot/tools/diagnose-workload.sh create mode 100755 cluster-troubleshoot/tools/discover-metrics.sh create mode 100755 cluster-troubleshoot/tools/fetch-alert-rule.sh create mode 100755 cluster-troubleshoot/tools/get-events.sh create mode 100755 cluster-troubleshoot/tools/get-firing-alerts.sh create mode 100755 cluster-troubleshoot/tools/prometheus-query-range.sh create mode 100755 cluster-troubleshoot/tools/prometheus-query.sh create mode 100755 cluster-troubleshoot/tools/prometheus-setup.sh create mode 100755 cluster-troubleshoot/tools/query-alert.sh create mode 100644 evals/skills/cluster-troubleshoot/system_prompt.md create mode 100644 evals/skills/cluster-troubleshoot/test_cases.yaml diff --git a/cluster-troubleshoot/README.md b/cluster-troubleshoot/README.md index c5d37c7..13fa339 100644 --- a/cluster-troubleshoot/README.md +++ b/cluster-troubleshoot/README.md @@ -1,3 +1,31 @@ -# Cluster Troubleshooting skills +# Cluster Troubleshooting Skill -This directory contains skills which are designed to help agents diagnose and troubleshoot OpenShift cluster issues. +Diagnose and troubleshoot OpenShift cluster issues: alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, and other cluster anomalies. + +## Structure + +``` +cluster-troubleshoot/ + SKILL.md # Skill definition (agent prompt) + tools/ # Diagnostic shell scripts (JSON output) +``` + +## Tools + +All scripts are in `tools/`, output JSON to stdout, and return structured error objects on failure. + +| Script | Purpose | +|--------|---------| +| `prometheus-setup.sh` | Get TOKEN and THANOS_URL for Prometheus access | +| `query-alert.sh` | Query firing instances of a named alert | +| `fetch-alert-rule.sh` | Get alert PromQL expression and annotations | +| `prometheus-query.sh` | Run an instant PromQL query | +| `prometheus-query-range.sh` | Run a range PromQL query | +| `discover-metrics.sh` | Search for available metric names | +| `get-firing-alerts.sh` | List all currently firing alerts | +| `diagnose-pod.sh` | Pod status, conditions, events, logs | +| `diagnose-node.sh` | Node conditions, capacity, taints | +| `diagnose-operator.sh` | ClusterOperator health | +| `diagnose-workload.sh` | Deployment/StatefulSet/DaemonSet status | +| `get-events.sh` | Events sorted by time | +| `check-recent-changes.sh` | Recent rollouts, image changes | diff --git a/cluster-troubleshoot/SKILL.md b/cluster-troubleshoot/SKILL.md index dc65030..7f8faf5 100644 --- a/cluster-troubleshoot/SKILL.md +++ b/cluster-troubleshoot/SKILL.md @@ -1,93 +1,145 @@ --- name: cluster-troubleshoot description: Diagnose and troubleshoot OpenShift cluster issues. Use when the user reports alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, HTTP errors, DNS failures, or any cluster anomaly. Not for cluster setup, configuration how-tos, or writing alerting rules. +allowed-tools: Bash --- -# ENVIRONMENT +# Environment - Platform: OpenShift Container Platform (OCP). Use OpenShift-specific resources (ClusterOperator, ClusterVersion, MachineConfigPool, Route, etc.) alongside standard Kubernetes ones. - You run commands against the live cluster. -- Available CLIs: oc, kubectl, jq, wget, openssl, skopeo, python3 -- Networking diagnostics: iproute (ip, ss), bind-utils (dig, nslookup, host), net-tools (netstat, ifconfig), tcpdump, lsof -- Process and system: procps-ng (ps, top, free), strace, findutils, file, diffutils -- Container images: skopeo -- General: git, less, vim, tar, gzip, unzip, diff +- Available CLIs: oc, kubectl, jq, wget, openssl, skopeo, python3, dig, nslookup, ip, ss, tcpdump, strace - Writable directories: /home/agent, /tmp/agent-workspace. The root filesystem is read-only. -# RESPONSE RULES +# Rules - Verify every claim with evidence from command output. Provide exact resource names, namespaces, timestamps, and error messages. - If multiple causes exist, list them numbered with supporting evidence. - If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information. -- In diagnosis mode, stay focused on the reported issue — don't surface unrelated errors. -- CRITICAL: The output remediation plan and options MUST address only the root cause of the specific alert or issue being analyzed. Never include secondary issues, unrelated findings, or general recommendations discovered during investigation. +- Stay focused on the reported issue — don't surface unrelated errors. +- The output remediation plan must address only the root cause of the specific alert or issue being analyzed. Never include secondary issues, unrelated findings, or general recommendations. - No URLs unless from command output or provided context. +- List actual resources before inspecting them — never guess pod names, use label selectors or owner references. +- Sample up to 3 representative pods per workload, not all. +- All `tools/` scripts output JSON. Use `jq` for further filtering when needed. +- Do not repeat the same command with the same arguments. +- Do not ask the user to run a command — gather the information yourself. +- Be highly concise. Evidence-backed conclusions, no filler. -# INVESTIGATION PROTOCOL +# Tools -When diagnosing a specific symptom: +The `tools/` directory contains diagnostic scripts. All output JSON to stdout and return structured error objects on failure. Run `eval $(bash tools/prometheus-setup.sh)` once per session before using Prometheus tools. -1. **Scope the blast radius** — is it one pod, one node, one namespace, or cluster-wide? This determines which layer to start from. +## Prometheus -2. **Gather evidence in parallel** — run independent `oc` commands together. Inspect the owner workload, pods, logs, events, services, and routes. Check pod conditions and container statuses, not just the phase — a pod can show "Running" but have failing health probes. +- `bash tools/prometheus-setup.sh` — Set TOKEN and THANOS_URL +- `bash tools/query-alert.sh ` — Firing instances of a named alert with full label sets +- `bash tools/fetch-alert-rule.sh ` — Alert PromQL expression, thresholds, annotations +- `bash tools/prometheus-query.sh ''` — Instant PromQL query +- `bash tools/prometheus-query-range.sh '' [duration] [step]` — Range query (default: 1h, 60s step) +- `bash tools/discover-metrics.sh ` — Search metric names by pattern +- `bash tools/get-firing-alerts.sh [filter]` — All firing alerts, optionally filtered by name -3. **Trace causality chains** — if resource A fails because of B, investigate B. Common chains in OpenShift: - - Pod pending → node pressure, unschedulable nodes, resource quota, or PVC not bound - - Pod crash-looping → OOMKilled (check limits vs. actual usage), application error (check logs), misconfigured probes - - Service unreachable → no endpoints → pods not ready → failing readiness probe - - Operator degraded → operand pod failing → node issue or config error - - Node NotReady → kubelet issues → disk pressure, certificate expired, MCO update stuck, or kernel panic +## Cluster -4. **Check recent changes** — many issues are caused by something that just changed. Compare symptom onset with rollout history, image tag changes, config/secret edits, HPA scaling, operator upgrades, node drains, and MachineConfig updates. Use `oc rollout history`, `oc describe`, and events sorted by time. +- `bash tools/diagnose-pod.sh [namespace] [--logs]` — Pod status, conditions, events, logs. Selectors (containing `=`) sample up to 3 pods. +- `bash tools/diagnose-node.sh [node_name]` — Node conditions, capacity, taints. Without a name: all nodes summary. +- `bash tools/diagnose-operator.sh [operator_name]` — ClusterOperator health. Without a name: all operators summary. +- `bash tools/diagnose-workload.sh [namespace] [kind]` — Deployment/StatefulSet/DaemonSet status, replicas, rollout history. Auto-detects kind. +- `bash tools/get-events.sh [namespace] [minutes]` — Events sorted by time, last N minutes (default: 60). Use `--all` for cluster-wide. +- `bash tools/check-recent-changes.sh [namespace] [minutes]` — Recent rollouts, image pulls, active rollouts. -5. **Keep digging** — after identifying a root cause, continue investigating to collect exact names, versions, and labels, and to check for additional contributing factors. +# Investigation Protocol -6. **Recommend a fix** if you know one, with the exact commands to run. Otherwise suggest mitigations and note whether each is reversible. +The protocol has two phases: **collect evidence first**, then **analyze**. Do not jump to conclusions or propose fixes until you have completed the collection phase. -# TOOL USAGE +## Phase 1 — Collect evidence -- List actual resources before inspecting them — never guess pod names, use label selectors or owner references. -- Sample up to 3 representative pods per workload, not all. -- Use `-o json | jq` for structured data extraction. -- Do not repeat the same command with the same arguments. -- Do not ask the user to run a command — gather the information yourself. +### Step 1 — Set up and identify the problem signal -# QUERYING PROMETHEUS +Configure Prometheus access: +```bash +eval $(bash tools/prometheus-setup.sh) +``` + +**If the user provides an alert name:** +```bash +bash tools/query-alert.sh +bash tools/fetch-alert-rule.sh +``` +If the alert is firing, extract the full label set (namespace, pod, node, service, severity, etc.). If it is NOT firing, still fetch the rule — the PromQL expression and labels contain diagnostic information. -OpenShift exposes Prometheus metrics through the Thanos Querier in the `openshift-monitoring` namespace. Query it via the external route, authenticating with your token. +If the rule is found, check the annotations for `runbook_url`. If present, extract the URL and follow the runbook steps as part of the analysis. -**Setup** — run once per session: +Run the alert's PromQL expression to see current and recent values: ```bash -TOKEN=$(oc whoami -t) -THANOS_URL=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}') +bash tools/prometheus-query.sh '' +bash tools/prometheus-query-range.sh '' ``` -**Instant query** (current value): +**If the user describes a symptom without an alert name** (e.g., "pods crashing in payments", "namespace X is broken"): ```bash -wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://$THANOS_URL/api/v1/query?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')" | jq . +bash tools/get-firing-alerts.sh ``` +Filter the output for alerts matching the namespace or resources the user mentioned. If alerts are found, pick the most relevant one and fetch its rule as above. If no alerts are firing, use metric exploration: +```bash +bash tools/discover-metrics.sh +bash tools/discover-metrics.sh +``` +Query discovered metrics with `bash tools/prometheus-query.sh` to look for anomalies. -**Range query** (time series, e.g. last hour with 60s resolution): +### Step 2 — Collect workload state and logs from all workloads + +List all deployments, statefulsets, and daemonsets in the affected namespace. Then for **every** workload, collect its status and logs: ```bash -wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://$THANOS_URL/api/v1/query_range?query=$(python3 -c 'import urllib.parse; print(urllib.parse.quote(""))')&start=$(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ)&end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&step=60s" | jq . +bash tools/diagnose-workload.sh +bash tools/diagnose-pod.sh --logs ``` +This is mandatory. Do not skip workloads that appear healthy. A Running pod with no visible issues can still be the root cause if it is producing errors internally. + +### Step 3 — Collect events and recent changes -**Discover available metrics** (never guess metric names): ```bash -wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://$THANOS_URL/api/v1/label/__name__/values" | jq '.data[]' | grep -i '' +bash tools/get-events.sh +bash tools/check-recent-changes.sh ``` -**Get firing alerts:** +### Step 4 — Collect firing alerts for correlation + ```bash -wget -qO- --no-check-certificate --header="Authorization: Bearer $TOKEN" \ - "https://$THANOS_URL/api/v1/alerts" | jq '.data.alerts[] | select(.state=="firing")' +bash tools/get-firing-alerts.sh ``` -**Workflow:** Start by checking firing alerts — their labels contain exact identifiers (namespace, pod, node) that make follow-up queries precise. Always discover metric names before querying. Use instant queries for current state, range queries for trends. If a metric doesn't exist, tell the user — do not fabricate PromQL. +## Phase 2 — Analyze and diagnose -# STYLE +### Step 5 — Correlate alerts -- Be highly concise. Evidence-backed conclusions, no filler. +Group all firing alerts by shared labels to find the common root cause: +- Alerts sharing the same **node** → likely a node-level issue (investigate the node first) +- Alerts sharing the same **namespace** but on different nodes → likely an application or config issue +- Alerts sharing the same **job** or **service** → dependency or networking issue +- Alerts across multiple namespaces and nodes → cluster-wide issue (check operators, control plane, networking) + +Prioritize investigation from infrastructure inward: node → operator → workload → pod. + +### Step 6 — Trace causality chains + +Use the evidence collected in phase 1 to trace the chain from symptom to root cause. If resource A fails because of B, investigate B. At each link, check status conditions and events. Common chains in OpenShift: +- Pod pending → pod conditions (PodScheduled, Unschedulable) and events → node pressure, unschedulable nodes, resource quota, or PVC not bound +- Pod crash-looping → container statuses (terminated reason: OOMKilled, Error) and logs → OOMKilled (check limits vs. actual usage), application error, misconfigured probes +- Service unreachable → endpoints/endpointslices → no endpoints → pods not ready → readiness probe and pod conditions +- Operator degraded → `bash tools/diagnose-operator.sh ` (check Available, Degraded, Progressing with messages) → operand pod failing → node issue or config error +- Node NotReady → node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and events → kubelet issues, certificate expired, MCO update stuck, or kernel panic + +Use the range query data from step 1 and the recent changes from step 3 to identify **when** the problem started and what changed around that time. + +### Step 7 — Recommend a fix + +Based on the collected evidence, propose a remediation. Provide the exact commands to run. When multiple options exist, list them from least to most disruptive and note whether each action is reversible. If the root cause is unclear, state what is known, suggest mitigations to reduce impact, and identify what additional data would narrow the diagnosis. + +**RBAC requirements for remediation:** the agent's ServiceAccount has **read-only** permissions (`cluster-reader` + `cluster-monitoring-view`). Any proposed remediation command (patch, delete, scale, rollout restart, etc.) requires additional RBAC that the agent does not have. For every proposed fix, you **must** include a complete RBAC section listing: +- The exact API group, resource, and verbs required (e.g., `apps` / `deployments` / `patch`) +- The namespace scope (specific namespace or cluster-wide) +- A ready-to-apply Role or ClusterRole YAML snippet the admin can use to grant the permissions + +This is critical — without this information the admin cannot safely execute the proposed fix. diff --git a/cluster-troubleshoot/tools/check-recent-changes.sh b/cluster-troubleshoot/tools/check-recent-changes.sh new file mode 100755 index 0000000..245a5bc --- /dev/null +++ b/cluster-troubleshoot/tools/check-recent-changes.sh @@ -0,0 +1,86 @@ +#!/bin/bash +# Detect recent changes that may have caused an issue: rollouts, image pulls, +# and scaling events. +# +# Usage: check-recent-changes.sh [namespace] [minutes] +# namespace: target namespace (default: current). Cluster-level changes always included. +# minutes: time window (default: 60) + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access + +NAMESPACE="${1:-}" +MINUTES="${2:-60}" + +NS_ARGS=() +if [[ -n "$NAMESPACE" ]]; then + NS_ARGS=(-n "$NAMESPACE") +fi + +# Deployment rollout status in namespace +DEPLOYMENTS="[]" +if [[ -n "$NAMESPACE" ]]; then + DEPLOY_JSON=$(oc get deployments "${NS_ARGS[@]}" -o json 2>/dev/null) || DEPLOY_JSON='{"items":[]}' + require_json "$DEPLOY_JSON" "deployment list" + DEPLOYMENTS=$(echo "$DEPLOY_JSON" | jq '[.items[] | select( + (.status.conditions[]? | select(.type=="Progressing" and .status=="True" and (.reason? == "NewReplicaSetAvailable" or .reason? == "ReplicaSetUpdated"))) + or (.status.unavailableReplicas // 0) > 0 + ) | { + name: .metadata.name, + namespace: .metadata.namespace, + ready: "\(.status.readyReplicas // 0)/\(.spec.replicas // 0)", + updated: (.status.updatedReplicas // 0), + unavailable: (.status.unavailableReplicas // 0), + last_update: (.status.conditions[] | select(.type=="Progressing") | .lastUpdateTime) + }]' 2>/dev/null || echo "[]") +fi + +CUTOFF=$(compute_cutoff "$MINUTES") + +# Fetch events once, filter into rollout and image-pull categories +ALL_EVENTS=$(oc get events "${NS_ARGS[@]}" --sort-by='.lastTimestamp' -o json 2>/dev/null) || \ + ALL_EVENTS='{"items":[]}' +require_json "$ALL_EVENTS" "events" + +ROLLOUT_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[ + .items[] + | select(.lastTimestamp >= $cutoff or .lastTimestamp == null) + | select(.reason | test("ScalingReplicaSet|DeploymentRollback|Scheduled|Pulled|Created|Started|Killing"; "i")) + | { + time: .lastTimestamp, + reason: .reason, + object: "\(.involvedObject.kind)/\(.involvedObject.name)", + namespace: .involvedObject.namespace, + message: .message + } + ]' 2>/dev/null || echo "[]") + +IMAGE_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[ + .items[] + | select(.lastTimestamp >= $cutoff or .lastTimestamp == null) + | select(.reason == "Pulled" or .reason == "Pulling") + | { + time: .lastTimestamp, + object: "\(.involvedObject.kind)/\(.involvedObject.name)", + namespace: .involvedObject.namespace, + message: .message + } + ]' 2>/dev/null || echo "[]") + +jq -n \ + --arg ns "${NAMESPACE:-cluster}" \ + --arg minutes "$MINUTES" \ + --argjson rollouts "$ROLLOUT_EVENTS" \ + --argjson images "$IMAGE_EVENTS" \ + --argjson deployments "$DEPLOYMENTS" \ + '{ + namespace: $ns, + time_window_minutes: ($minutes | tonumber), + rollout_events: $rollouts, + image_pull_events: $images, + active_rollouts: $deployments + }' diff --git a/cluster-troubleshoot/tools/common.sh b/cluster-troubleshoot/tools/common.sh new file mode 100755 index 0000000..8902df8 --- /dev/null +++ b/cluster-troubleshoot/tools/common.sh @@ -0,0 +1,125 @@ +#!/bin/bash +# Shared functions for cluster-troubleshoot tools. +# Source this file: . "$(dirname "$0")/common.sh" + +set -euo pipefail + +# --- JSON output helpers --- + +error_json() { + local code="$1" + local message="$2" + local suggestion="${3:-}" + if [[ -n "$suggestion" ]]; then + jq -n --arg c "$code" --arg m "$message" --arg s "$suggestion" \ + '{error:true, code:$c, message:$m, suggestion:$s}' + else + jq -n --arg c "$code" --arg m "$message" \ + '{error:true, code:$c, message:$m}' + fi + exit 1 +} + +# --- Argument and environment validation --- + +require_command() { + local cmd="$1" + if ! command -v "$cmd" &>/dev/null; then + error_json "MISSING_TOOL" "$cmd is not available on PATH" "Install $cmd or verify the container image includes it" + fi +} + +require_env() { + local var_name="$1" + if [[ -z "${!var_name:-}" ]]; then + error_json "NOT_CONFIGURED" "$var_name is not set" "Run: eval \$(bash tools/prometheus-setup.sh)" + fi +} + +require_arg() { + local value="$1" + local name="$2" + local usage="${3:-}" + if [[ -z "$value" ]]; then + if [[ -n "$usage" ]]; then + error_json "MISSING_ARG" "$name is required" "$usage" + else + error_json "MISSING_ARG" "$name is required" + fi + fi +} + +# --- Cluster access --- + +check_cluster_access() { + if ! oc whoami &>/dev/null; then + error_json "AUTH_FAILED" "Cannot authenticate to cluster" "Run: oc login " + fi +} + +# --- URL encoding --- + +urlencode() { + printf '%s' "$1" | python3 -c "import sys, urllib.parse; print(urllib.parse.quote(sys.stdin.read()))" +} + +# --- Date helpers --- + +compute_cutoff() { + local minutes="$1" + date -u -d "${minutes} minutes ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || \ + date -u -d "@$(($(date +%s) - minutes * 60))" +%Y-%m-%dT%H:%M:%SZ +} + +# --- JSON validation --- + +require_json() { + local input="$1" + local context="${2:-API response}" + if ! echo "$input" | jq empty 2>/dev/null; then + error_json "INVALID_JSON" "Failed to parse ${context} as JSON" \ + "The cluster API returned unexpected output" + fi +} + +# --- Prometheus helpers --- + +prometheus_get() { + local path="$1" + require_env TOKEN + require_env THANOS_URL + + local http_code + local tmpfile + local stderr_file + tmpfile=$(mktemp) + stderr_file=$(mktemp) + trap 'rm -f "$tmpfile" "$stderr_file"' RETURN + + wget -qO "$tmpfile" --no-check-certificate \ + --header="Authorization: Bearer $TOKEN" \ + --server-response \ + "https://${THANOS_URL}${path}" 2>"$stderr_file" || true + + http_code=$(grep "HTTP/" "$stderr_file" 2>/dev/null | tail -1 | awk '{print $2}') || true + + if [[ ! -s "$tmpfile" ]]; then + error_json "PROMETHEUS_ERROR" "No response from Prometheus at ${THANOS_URL}${path}" \ + "Verify TOKEN and THANOS_URL with: eval \$(bash tools/prometheus-setup.sh)" + fi + + if [[ -n "$http_code" && "$http_code" -ge 400 ]] 2>/dev/null; then + local body + body=$(cat "$tmpfile") + case "$http_code" in + 401) error_json "AUTH_EXPIRED" "Prometheus returned 401 Unauthorized" \ + "Token may have expired. Re-run: eval \$(bash tools/prometheus-setup.sh)" ;; + 403) error_json "FORBIDDEN" "Prometheus returned 403 Forbidden" \ + "ServiceAccount lacks cluster-monitoring-view ClusterRoleBinding" ;; + *) error_json "PROMETHEUS_HTTP_${http_code}" "Prometheus returned HTTP ${http_code}" \ + "Response: ${body:0:200}" ;; + esac + fi + + cat "$tmpfile" +} diff --git a/cluster-troubleshoot/tools/diagnose-node.sh b/cluster-troubleshoot/tools/diagnose-node.sh new file mode 100755 index 0000000..6bc4ca3 --- /dev/null +++ b/cluster-troubleshoot/tools/diagnose-node.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# Node diagnostics: conditions, capacity, allocatable, taints, kubelet info. +# +# Usage: diagnose-node.sh [node_name] +# If no node name given, shows all nodes with non-Ready conditions. + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access + +NODE_NAME="${1:-}" + +if [[ -n "$NODE_NAME" ]]; then + NODE_JSON=$(oc get node "$NODE_NAME" -o json 2>/dev/null) || \ + error_json "NOT_FOUND" "Node '$NODE_NAME' not found" "List nodes with: oc get nodes" + require_json "$NODE_JSON" "node $NODE_NAME" + + echo "$NODE_JSON" | jq '{ + nodes: [{ + name: .metadata.name, + labels: (.metadata.labels | with_entries(select(.key | test("node-role|kubernetes.io/os|topology")))), + conditions: .status.conditions, + capacity: .status.capacity, + allocatable: .status.allocatable, + taints: (.spec.taints // []), + unschedulable: (.spec.unschedulable // false), + kubelet_version: .status.nodeInfo.kubeletVersion, + os_image: .status.nodeInfo.osImage, + container_runtime: .status.nodeInfo.containerRuntimeVersion + }] + }' +else + # Show all nodes, flagging unhealthy ones + NODES_JSON=$(oc get nodes -o json 2>/dev/null) || \ + error_json "CLUSTER_ERROR" "Cannot list nodes" + require_json "$NODES_JSON" "node list" + + echo "$NODES_JSON" | jq '{ + nodes: [.items[] | { + name: .metadata.name, + labels: (.metadata.labels | with_entries(select(.key | test("node-role|kubernetes.io/os|topology")))), + ready: (.status.conditions[] | select(.type=="Ready") | .status == "True"), + conditions: [.status.conditions[] | select(.status != "False" or .type == "Ready")], + taints: (.spec.taints // []), + unschedulable: (.spec.unschedulable // false), + kubelet_version: .status.nodeInfo.kubeletVersion + }], + summary: { + total: (.items | length), + ready: [.items[] | select(.status.conditions[] | select(.type=="Ready" and .status=="True"))] | length, + not_ready: [.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True"))] | length + } + }' +fi diff --git a/cluster-troubleshoot/tools/diagnose-operator.sh b/cluster-troubleshoot/tools/diagnose-operator.sh new file mode 100755 index 0000000..b899abf --- /dev/null +++ b/cluster-troubleshoot/tools/diagnose-operator.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# ClusterOperator diagnostics: Available, Progressing, Degraded conditions. +# +# Usage: diagnose-operator.sh [operator_name] +# If no operator name given, lists all operators with non-healthy conditions. + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access + +OPERATOR_NAME="${1:-}" + +if [[ -n "$OPERATOR_NAME" ]]; then + CO_JSON=$(oc get clusteroperator "$OPERATOR_NAME" -o json 2>/dev/null) || \ + error_json "NOT_FOUND" "ClusterOperator '$OPERATOR_NAME' not found" "List operators with: oc get clusteroperators" + require_json "$CO_JSON" "ClusterOperator $OPERATOR_NAME" + + echo "$CO_JSON" | jq '{ + operators: [{ + name: .metadata.name, + available: (.status.conditions[] | select(.type=="Available") | .status == "True"), + progressing: (.status.conditions[] | select(.type=="Progressing") | .status == "True"), + degraded: (.status.conditions[] | select(.type=="Degraded") | .status == "True"), + versions: [.status.versions[]? | {(.name): .version}] | add // {}, + conditions: .status.conditions + }] + }' +else + CO_JSON=$(oc get clusteroperators -o json 2>/dev/null) || \ + error_json "CLUSTER_ERROR" "Cannot list ClusterOperators" + require_json "$CO_JSON" "ClusterOperator list" + + echo "$CO_JSON" | jq '{ + operators: [.items[] | { + name: .metadata.name, + available: (.status.conditions[] | select(.type=="Available") | .status == "True"), + progressing: (.status.conditions[] | select(.type=="Progressing") | .status == "True"), + degraded: (.status.conditions[] | select(.type=="Degraded") | .status == "True"), + versions: [.status.versions[]? | {(.name): .version}] | add // {} + }], + summary: { + total: (.items | length), + available: [.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="True"))] | length, + degraded: [.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True"))] | length, + progressing: [.items[] | select(.status.conditions[] | select(.type=="Progressing" and .status=="True"))] | length + } + }' +fi diff --git a/cluster-troubleshoot/tools/diagnose-pod.sh b/cluster-troubleshoot/tools/diagnose-pod.sh new file mode 100755 index 0000000..f576775 --- /dev/null +++ b/cluster-troubleshoot/tools/diagnose-pod.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# Pod diagnostics: status, conditions, container statuses, events, and logs. +# +# Usage: diagnose-pod.sh [namespace] [--logs] +# If contains '=', it is treated as a label selector. +# Samples up to 3 pods when using a label selector. + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access +require_arg "${1:-}" "pod name or label selector" "Usage: diagnose-pod.sh [namespace] [--logs]" + +TARGET="$1" +NAMESPACE="${2:-}" +INCLUDE_LOGS=false + +# Parse flags from remaining args +for arg in "$@"; do + if [[ "$arg" == "--logs" ]]; then + INCLUDE_LOGS=true + fi +done + +NS_ARGS=() +if [[ -n "$NAMESPACE" && "$NAMESPACE" != "--logs" ]]; then + NS_ARGS=(-n "$NAMESPACE") +fi + +# Determine if target is a label selector or pod name +if [[ "$TARGET" == *"="* ]]; then + POD_JSON=$(oc get pods -l "$TARGET" "${NS_ARGS[@]}" -o json 2>/dev/null) || \ + error_json "NOT_FOUND" "No pods matching selector '$TARGET'" "Verify the label selector and namespace" + require_json "$POD_JSON" "pod list" + POD_NAMES=$(echo "$POD_JSON" | jq -r '.items[].metadata.name' | head -3) +else + POD_JSON=$(oc get pod "$TARGET" "${NS_ARGS[@]}" -o json 2>/dev/null) || \ + error_json "NOT_FOUND" "Pod '$TARGET' not found" "Check the pod name and namespace" + require_json "$POD_JSON" "pod" + POD_NAMES="$TARGET" +fi + +if [[ -z "$POD_NAMES" ]]; then + error_json "NOT_FOUND" "No pods found matching '$TARGET'" +fi + +RESULTS="[]" + +while IFS= read -r POD_NAME; do + [[ -z "$POD_NAME" ]] && continue + + # Get pod details + POD_DETAIL=$(oc get pod "$POD_NAME" "${NS_ARGS[@]}" -o json 2>/dev/null) || continue + require_json "$POD_DETAIL" "pod $POD_NAME" + POD_NS=$(echo "$POD_DETAIL" | jq -r '.metadata.namespace') + + # Extract key fields + POD_INFO=$(echo "$POD_DETAIL" | jq '{ + name: .metadata.name, + namespace: .metadata.namespace, + phase: .status.phase, + conditions: .status.conditions, + container_statuses: .status.containerStatuses, + init_container_statuses: .status.initContainerStatuses, + owner_references: .metadata.ownerReferences, + node_name: .spec.nodeName, + restart_count: ([.status.containerStatuses[]?.restartCount] | add // 0) + }') + + # Get events for this pod + EVENTS=$(oc get events -n "$POD_NS" \ + --field-selector "involvedObject.name=$POD_NAME" \ + --sort-by='.lastTimestamp' -o json 2>/dev/null | \ + jq '[.items[-10:] | .[] | { + time: .lastTimestamp, + type: .type, + reason: .reason, + message: .message, + count: .count + }]' 2>/dev/null || echo "[]") + + POD_INFO=$(echo "$POD_INFO" | jq --argjson events "$EVENTS" '. + {events: $events}') + + # Get logs if requested + if [[ "$INCLUDE_LOGS" == "true" ]]; then + CONTAINERS=$(echo "$POD_DETAIL" | jq -r '.spec.containers[].name') + LOGS_OBJ="{}" + while IFS= read -r CONTAINER; do + [[ -z "$CONTAINER" ]] && continue + CONTAINER_LOGS=$(oc logs "$POD_NAME" -n "$POD_NS" -c "$CONTAINER" --tail=50 2>/dev/null || echo "(logs unavailable)") + LOGS_ARRAY=$(echo "$CONTAINER_LOGS" | jq -R . | jq -s .) + LOGS_OBJ=$(echo "$LOGS_OBJ" | jq --arg c "$CONTAINER" --argjson l "$LOGS_ARRAY" '. + {($c): $l}') + done <<< "$CONTAINERS" + POD_INFO=$(echo "$POD_INFO" | jq --argjson logs "$LOGS_OBJ" '. + {logs: $logs}') + fi + + RESULTS=$(echo "$RESULTS" | jq --argjson pod "$POD_INFO" '. + [$pod]') +done <<< "$POD_NAMES" + +echo "$RESULTS" | jq '{pods: .}' diff --git a/cluster-troubleshoot/tools/diagnose-workload.sh b/cluster-troubleshoot/tools/diagnose-workload.sh new file mode 100755 index 0000000..48baedc --- /dev/null +++ b/cluster-troubleshoot/tools/diagnose-workload.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# Workload diagnostics: Deployment, StatefulSet, or DaemonSet status, +# replica counts, conditions, and rollout history. +# +# Usage: diagnose-workload.sh [namespace] [kind] +# kind: deployment, statefulset, daemonset (default: auto-detect) + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access +require_arg "${1:-}" "workload name" "Usage: diagnose-workload.sh [namespace] [kind]" + +WORKLOAD_NAME="$1" +NAMESPACE="${2:-}" +KIND="${3:-}" + +NS_ARGS=() +if [[ -n "$NAMESPACE" ]]; then + NS_ARGS=(-n "$NAMESPACE") +fi + +# Auto-detect kind if not specified +if [[ -z "$KIND" ]]; then + for try_kind in deployment statefulset daemonset; do + if oc get "$try_kind" "$WORKLOAD_NAME" "${NS_ARGS[@]}" &>/dev/null; then + KIND="$try_kind" + break + fi + done + if [[ -z "$KIND" ]]; then + error_json "NOT_FOUND" "No deployment, statefulset, or daemonset named '$WORKLOAD_NAME' found" \ + "Verify the name and namespace" + fi +fi + +WL_JSON=$(oc get "$KIND" "$WORKLOAD_NAME" "${NS_ARGS[@]}" -o json 2>/dev/null) || \ + error_json "NOT_FOUND" "$KIND '$WORKLOAD_NAME' not found" +require_json "$WL_JSON" "$KIND $WORKLOAD_NAME" + +# Get rollout history +HISTORY=$(oc rollout history "$KIND/$WORKLOAD_NAME" "${NS_ARGS[@]}" 2>/dev/null || echo "") + +# Build output based on kind +case "$KIND" in + deployment) + echo "$WL_JSON" | jq --arg hist "$HISTORY" '{ + kind: "Deployment", + name: .metadata.name, + namespace: .metadata.namespace, + replicas: { + desired: (.spec.replicas // 0), + ready: (.status.readyReplicas // 0), + available: (.status.availableReplicas // 0), + updated: (.status.updatedReplicas // 0), + unavailable: (.status.unavailableReplicas // 0) + }, + strategy: .spec.strategy.type, + conditions: .status.conditions, + containers: [.spec.template.spec.containers[] | {name, image}], + rollout_history: $hist + }' + ;; + statefulset) + echo "$WL_JSON" | jq --arg hist "$HISTORY" '{ + kind: "StatefulSet", + name: .metadata.name, + namespace: .metadata.namespace, + replicas: { + desired: (.spec.replicas // 0), + ready: (.status.readyReplicas // 0), + current: (.status.currentReplicas // 0), + updated: (.status.updatedReplicas // 0) + }, + update_strategy: .spec.updateStrategy.type, + conditions: .status.conditions, + containers: [.spec.template.spec.containers[] | {name, image}], + volume_claims: [.spec.volumeClaimTemplates[]? | .metadata.name], + rollout_history: $hist + }' + ;; + daemonset) + echo "$WL_JSON" | jq --arg hist "$HISTORY" '{ + kind: "DaemonSet", + name: .metadata.name, + namespace: .metadata.namespace, + replicas: { + desired: (.status.desiredNumberScheduled // 0), + ready: (.status.numberReady // 0), + available: (.status.numberAvailable // 0), + misscheduled: (.status.numberMisscheduled // 0), + unavailable: (.status.numberUnavailable // 0) + }, + update_strategy: .spec.updateStrategy.type, + conditions: .status.conditions, + containers: [.spec.template.spec.containers[] | {name, image}], + rollout_history: $hist + }' + ;; +esac diff --git a/cluster-troubleshoot/tools/discover-metrics.sh b/cluster-troubleshoot/tools/discover-metrics.sh new file mode 100755 index 0000000..c901df1 --- /dev/null +++ b/cluster-troubleshoot/tools/discover-metrics.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# Search for Prometheus metric names matching a pattern. +# +# Usage: discover-metrics.sh + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 +require_arg "${1:-}" "search pattern" "Usage: discover-metrics.sh " + +PATTERN="$1" + +ALL_METRICS=$(prometheus_get "/api/v1/label/__name__/values") +MATCHED=$(echo "$ALL_METRICS" | jq -r '.data[]' | grep -i "$PATTERN" || true) + +if [[ -z "$MATCHED" ]]; then + jq -n --arg p "$PATTERN" '{pattern: $p, match_count: 0, metrics: []}' +else + echo "$MATCHED" | jq -R . | jq -s --arg p "$PATTERN" \ + '{pattern: $p, match_count: length, metrics: .}' +fi diff --git a/cluster-troubleshoot/tools/fetch-alert-rule.sh b/cluster-troubleshoot/tools/fetch-alert-rule.sh new file mode 100755 index 0000000..9e431e0 --- /dev/null +++ b/cluster-troubleshoot/tools/fetch-alert-rule.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Fetch the alerting rule definition: PromQL expression, thresholds, +# duration, labels, and annotations (description, summary, runbook_url). +# +# Usage: fetch-alert-rule.sh + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 +require_arg "${1:-}" "alert name" "Usage: fetch-alert-rule.sh " + +ALERT_NAME="$1" + +RESPONSE=$(prometheus_get "/api/v1/rules?type=alert") +RULE=$(echo "$RESPONSE" | jq --arg name "$ALERT_NAME" \ + '[.data.groups[].rules[] | select(.name==$name and .type=="alerting")] | first // empty') + +if [[ -z "$RULE" || "$RULE" == "null" ]]; then + printf '{"alert_name":"%s","found":false}\n' "$ALERT_NAME" +else + echo "$RULE" | jq --arg name "$ALERT_NAME" '{ + alert_name: $name, + found: true, + expr: .query, + duration: .duration, + labels: .labels, + annotations: .annotations + }' +fi diff --git a/cluster-troubleshoot/tools/get-events.sh b/cluster-troubleshoot/tools/get-events.sh new file mode 100755 index 0000000..d95bf08 --- /dev/null +++ b/cluster-troubleshoot/tools/get-events.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Get events sorted by time for a namespace or cluster-wide. +# +# Usage: get-events.sh [namespace] [minutes] +# namespace: target namespace, or '--all' for cluster-wide (default: current) +# minutes: only show events from last N minutes (default: 60) + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access + +NAMESPACE="${1:-}" +MINUTES="${2:-60}" + +NS_ARGS=() +if [[ "$NAMESPACE" == "--all" || "$NAMESPACE" == "-A" ]]; then + NS_ARGS=(-A) + NAMESPACE="all" +elif [[ -n "$NAMESPACE" ]]; then + NS_ARGS=(-n "$NAMESPACE") +fi + +CUTOFF=$(compute_cutoff "$MINUTES") + +EVENTS_JSON=$(oc get events "${NS_ARGS[@]}" --sort-by='.lastTimestamp' -o json 2>/dev/null) || \ + error_json "CLUSTER_ERROR" "Cannot list events" +require_json "$EVENTS_JSON" "events" + +# Filter to recent events and extract key fields +echo "$EVENTS_JSON" | jq --arg cutoff "$CUTOFF" --arg ns "${NAMESPACE:-current}" --argjson minutes "$MINUTES" '{ + namespace: $ns, + time_window_minutes: $minutes, + events: [ + .items[] + | select(.lastTimestamp >= $cutoff or .lastTimestamp == null) + | { + time: .lastTimestamp, + type: .type, + reason: .reason, + object: "\(.involvedObject.kind)/\(.involvedObject.name)", + namespace: .involvedObject.namespace, + message: .message, + count: .count + } + ] | .[-100:], + total_in_window: ([ + .items[] + | select(.lastTimestamp >= $cutoff or .lastTimestamp == null) + ] | length) +}' diff --git a/cluster-troubleshoot/tools/get-firing-alerts.sh b/cluster-troubleshoot/tools/get-firing-alerts.sh new file mode 100755 index 0000000..5cc761d --- /dev/null +++ b/cluster-troubleshoot/tools/get-firing-alerts.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# List all currently firing alerts from Prometheus. +# +# Usage: get-firing-alerts.sh [filter] +# filter: optional grep pattern to filter alert names + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 + +FILTER="${1:-}" + +RESPONSE=$(prometheus_get "/api/v1/alerts") + +if [[ -n "$FILTER" ]]; then + ALERTS=$(echo "$RESPONSE" | jq --arg f "$FILTER" \ + '[.data.alerts[] | select(.state=="firing") | select(.labels.alertname | test($f; "i"))]') +else + ALERTS=$(echo "$RESPONSE" | jq '[.data.alerts[] | select(.state=="firing")]') +fi + +COUNT=$(echo "$ALERTS" | jq 'length') +printf '{"total_firing":%d,"alerts":%s}\n' "$COUNT" "$ALERTS" diff --git a/cluster-troubleshoot/tools/prometheus-query-range.sh b/cluster-troubleshoot/tools/prometheus-query-range.sh new file mode 100755 index 0000000..0f92946 --- /dev/null +++ b/cluster-troubleshoot/tools/prometheus-query-range.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# Run a PromQL range query with configurable time window and step. +# +# Usage: prometheus-query-range.sh '' [duration] [step] +# duration: lookback period (default: 1h). Supports: Ns, Nm, Nh, Nd +# step: resolution (default: 60s) + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 +require_arg "${1:-}" "PromQL expression" "Usage: prometheus-query-range.sh '' [duration] [step]" + +PROMQL="$1" +DURATION="${2:-1h}" +STEP="${3:-60s}" + +# Parse duration to seconds +parse_duration() { + local val="$1" + local num="${val%[smhd]}" + local unit="${val##*[0-9]}" + case "$unit" in + s) echo "$num" ;; + m) echo $((num * 60)) ;; + h) echo $((num * 3600)) ;; + d) echo $((num * 86400)) ;; + *) error_json "INVALID_ARG" "Invalid duration format: $val" "Use: Ns, Nm, Nh, or Nd (e.g. 30m, 1h, 1d)" ;; + esac +} + +SECONDS_AGO=$(parse_duration "$DURATION") +END=$(date -u +%Y-%m-%dT%H:%M:%SZ) +START=$(date -u -d "${SECONDS_AGO} seconds ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null) || \ + START=$(date -u -d "@$(($(date +%s) - SECONDS_AGO))" +%Y-%m-%dT%H:%M:%SZ) + +QUERY=$(urlencode "$PROMQL") +prometheus_get "/api/v1/query_range?query=${QUERY}&start=${START}&end=${END}&step=${STEP}" | jq . diff --git a/cluster-troubleshoot/tools/prometheus-query.sh b/cluster-troubleshoot/tools/prometheus-query.sh new file mode 100755 index 0000000..76c0ee4 --- /dev/null +++ b/cluster-troubleshoot/tools/prometheus-query.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# Run an instant PromQL query (current value). +# +# Usage: prometheus-query.sh '' + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 +require_arg "${1:-}" "PromQL expression" "Usage: prometheus-query.sh ''" + +QUERY=$(urlencode "$1") +prometheus_get "/api/v1/query?query=${QUERY}" | jq . diff --git a/cluster-troubleshoot/tools/prometheus-setup.sh b/cluster-troubleshoot/tools/prometheus-setup.sh new file mode 100755 index 0000000..17a74c9 --- /dev/null +++ b/cluster-troubleshoot/tools/prometheus-setup.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# Establish Prometheus access credentials for the session. +# Prints export commands to stdout for eval. +# +# Usage: eval $(bash tools/prometheus-setup.sh) + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command oc +require_command jq +check_cluster_access + +TOKEN=$(oc whoami -t 2>/dev/null) || \ + error_json "AUTH_FAILED" "Cannot get bearer token" "Run: oc login " + +THANOS_URL=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}' 2>/dev/null) || \ + error_json "ROUTE_NOT_FOUND" "thanos-querier route not found in openshift-monitoring" \ + "Verify the openshift-monitoring namespace exists and the route is exposed" + +if [[ -z "$THANOS_URL" ]]; then + error_json "ROUTE_NOT_FOUND" "thanos-querier route has no host" \ + "Check: oc get route thanos-querier -n openshift-monitoring" +fi + +printf 'export TOKEN=%s\n' "$TOKEN" +printf 'export THANOS_URL=%s\n' "$THANOS_URL" diff --git a/cluster-troubleshoot/tools/query-alert.sh b/cluster-troubleshoot/tools/query-alert.sh new file mode 100755 index 0000000..675e292 --- /dev/null +++ b/cluster-troubleshoot/tools/query-alert.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# Query Prometheus for firing instances of a specific alert. +# Returns the full label set for each firing instance. +# +# Usage: query-alert.sh + +SCRIPT_DIR="$(dirname "$0")" +. "$SCRIPT_DIR/common.sh" + +require_command jq +require_command python3 +require_arg "${1:-}" "alert name" "Usage: query-alert.sh " + +ALERT_NAME="$1" +PROMQL="ALERTS{alertname=\"${ALERT_NAME}\"}" +QUERY=$(urlencode "$PROMQL") + +RESPONSE=$(prometheus_get "/api/v1/query?query=${QUERY}") + +echo "$RESPONSE" | jq --arg name "$ALERT_NAME" '{ + alert_name: $name, + firing: (.data.result | length > 0), + instance_count: (.data.result | length), + instances: .data.result +}' diff --git a/evals/skills/cluster-troubleshoot/system_prompt.md b/evals/skills/cluster-troubleshoot/system_prompt.md new file mode 100644 index 0000000..41d5c09 --- /dev/null +++ b/evals/skills/cluster-troubleshoot/system_prompt.md @@ -0,0 +1,18 @@ +You are a cluster diagnostics assistant with access to the 'cluster-troubleshoot' skill. This skill provides shell scripts under tools/ for diagnosing OpenShift cluster issues: pod crashes, node failures, operator degradation, firing alerts, and other anomalies. + +Available diagnostic scripts (all under tools/): +- prometheus-setup.sh — Set TOKEN and THANOS_URL for Prometheus access +- query-alert.sh — Firing instances of a named alert +- fetch-alert-rule.sh — Alert PromQL expression, thresholds, annotations +- prometheus-query.sh '' — Instant PromQL query +- prometheus-query-range.sh '' [duration] [step] — Range query +- discover-metrics.sh — Search metric names by pattern +- get-firing-alerts.sh [filter] — All firing alerts +- diagnose-pod.sh [namespace] [--logs] — Pod status, conditions, events, logs +- diagnose-node.sh [node_name] — Node conditions, capacity, taints +- diagnose-operator.sh [operator_name] — ClusterOperator health +- diagnose-workload.sh [namespace] [kind] — Deployment/StatefulSet/DaemonSet status +- get-events.sh [namespace] [minutes] — Events sorted by time +- check-recent-changes.sh [namespace] [minutes] — Recent rollouts, image pulls + +The investigation protocol has two phases: collect evidence first (set up Prometheus, identify problem signal, collect workload state and logs, collect events and recent changes, collect firing alerts), then analyze (correlate alerts, trace causality chains, recommend a fix). diff --git a/evals/skills/cluster-troubleshoot/test_cases.yaml b/evals/skills/cluster-troubleshoot/test_cases.yaml new file mode 100644 index 0000000..3182a81 --- /dev/null +++ b/evals/skills/cluster-troubleshoot/test_cases.yaml @@ -0,0 +1,167 @@ +# cluster-troubleshoot skill eval test cases +# +# Static tests verify tool-selection knowledge — the agent must pick the +# correct diagnostic script for a given scenario. +# +# Mock-based tests embed canned tool output in the query and verify the +# agent interprets the diagnostic data correctly. + +# --- Static tool-selection tests --- + +- name: tool_for_crashing_pod + query: "A pod named web-frontend-7b9d4 is crash-looping in namespace production. Which cluster-troubleshoot diagnostic script should I run first to inspect its status, conditions, and events?" + schema: + type: object + properties: + script: + type: string + enum: ["diagnose-pod.sh", "diagnose-workload.sh", "diagnose-node.sh", "get-events.sh", "get-firing-alerts.sh"] + description: "The diagnostic script from the cluster-troubleshoot skill to run first. Pick the most specific one." + required: [script] + expected: + script: "diagnose-pod.sh" + +- name: tool_for_firing_alert + query: "An alert called KubePodCrashLooping is firing. Which cluster-troubleshoot diagnostic script fetches the alert's PromQL rule definition, thresholds, and annotations?" + schema: + type: object + properties: + script: + type: string + enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"] + description: "The script that retrieves the alert rule definition. Use the 'cluster-troubleshoot' skill to find this." + required: [script] + expected: + script: "fetch-alert-rule.sh" + +- name: tool_for_node_issue + query: "A node is showing NotReady status. Which cluster-troubleshoot diagnostic script provides node conditions, capacity, taints, and kubelet info?" + schema: + type: object + properties: + script: + type: string + enum: ["diagnose-node.sh", "diagnose-pod.sh", "diagnose-operator.sh", "get-events.sh", "check-recent-changes.sh"] + description: "The script for node diagnostics. Use the 'cluster-troubleshoot' skill to find this." + required: [script] + expected: + script: "diagnose-node.sh" + +- name: tool_for_operator_degraded + query: "The authentication ClusterOperator is showing Degraded=True. Which cluster-troubleshoot diagnostic script checks ClusterOperator health, conditions, and versions?" + schema: + type: object + properties: + script: + type: string + enum: ["diagnose-operator.sh", "diagnose-workload.sh", "diagnose-pod.sh", "get-firing-alerts.sh", "check-recent-changes.sh"] + description: "The script for ClusterOperator diagnostics. Use the 'cluster-troubleshoot' skill to find this." + required: [script] + expected: + script: "diagnose-operator.sh" + +- name: tool_for_prometheus_setup + query: "Before running any Prometheus queries with the cluster-troubleshoot skill, what script must be run first to configure access credentials?" + schema: + type: object + properties: + script: + type: string + enum: ["prometheus-setup.sh", "prometheus-query.sh", "discover-metrics.sh", "get-firing-alerts.sh"] + description: "The script that sets up Prometheus credentials. Use the 'cluster-troubleshoot' skill to find this." + required: [script] + expected: + script: "prometheus-setup.sh" + +- name: tool_for_recent_changes + query: "Something changed recently and broke our namespace. Which cluster-troubleshoot diagnostic script detects recent rollouts, image pulls, and scaling events?" + schema: + type: object + properties: + script: + type: string + enum: ["check-recent-changes.sh", "get-events.sh", "diagnose-workload.sh", "get-firing-alerts.sh", "diagnose-pod.sh"] + description: "The script for detecting recent cluster changes. Use the 'cluster-troubleshoot' skill to find this." + required: [script] + expected: + script: "check-recent-changes.sh" + +- name: diagnose_pod_log_flag + query: "When using the cluster-troubleshoot skill's diagnose-pod.sh, what command-line flag do you pass to include container logs in the output?" + schema: + type: object + properties: + flag: + type: string + enum: ["--logs", "--include-logs", "-l", "--verbose", "--tail"] + description: "The flag for including logs. Use the 'cluster-troubleshoot' skill to find this." + required: [flag] + expected: + flag: "--logs" + +- name: first_investigation_step + query: "According to the cluster-troubleshoot investigation protocol, what is the very first command you must run in Step 1 before anything else?" + schema: + type: object + properties: + command: + type: string + enum: + - "eval $(bash tools/prometheus-setup.sh)" + - "bash tools/get-firing-alerts.sh" + - "bash tools/diagnose-pod.sh" + - "bash tools/get-events.sh" + description: "The first command in the investigation protocol. Use the 'cluster-troubleshoot' skill to find this." + required: [command] + expected: + command: "eval $(bash tools/prometheus-setup.sh)" + +# --- Mock-based diagnostic reasoning tests --- + +- name: interpret_pod_oomkilled + query: | + You ran diagnose-pod.sh and got this output: + {"pods":[{"name":"api-server-6f8b9c-xk2lm","namespace":"payments","phase":"Running","conditions":[{"type":"Ready","status":"False"},{"type":"ContainersReady","status":"False"}],"container_statuses":[{"name":"api","state":{"waiting":{"reason":"CrashLoopBackOff"}},"lastState":{"terminated":{"reason":"OOMKilled","exitCode":137,"finishedAt":"2026-07-17T08:15:00Z"}},"restartCount":12,"ready":false}],"node_name":"worker-2","restart_count":12,"events":[{"type":"Warning","reason":"BackOff","message":"Back-off restarting failed container api in pod api-server-6f8b9c-xk2lm_payments"}]}]} + What is the root cause of this pod's failure? + schema: + type: object + properties: + root_cause: + type: string + enum: ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff", "Evicted", "NodeNotReady"] + description: "The underlying cause of the container's termination. Use the 'cluster-troubleshoot' skill to diagnose." + pod_name: + type: string + description: "The name of the affected pod" + restart_count: + type: integer + description: "Number of container restarts" + required: [root_cause, pod_name, restart_count] + expected: + root_cause: "OOMKilled" + pod_name: "api-server-6f8b9c-xk2lm" + restart_count: 12 + +- name: interpret_firing_alert + query: | + You ran query-alert.sh KubeDeploymentReplicasMismatch and got: + {"alert_name":"KubeDeploymentReplicasMismatch","firing":true,"instance_count":2,"instances":[{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"payment-gateway","severity":"warning"},"value":[1721203200,"1"]},{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"order-service","severity":"warning"},"value":[1721203200,"1"]}]} + Based on this output: is the alert firing, how many instances are affected, and in which namespace? + schema: + type: object + properties: + is_firing: + type: boolean + description: "Whether the alert is currently firing" + instance_count: + type: integer + description: "Number of firing instances" + namespace: + type: string + enum: ["checkout", "payments", "production", "default", "openshift-monitoring"] + description: "The namespace where the alert is firing" + required: [is_firing, instance_count, namespace] + expected: + is_firing: true + instance_count: 2 + namespace: "checkout" From 0ccc292e672c657345cfdf00b53acc57c3659b1c Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Tue, 21 Jul 2026 22:54:10 +0200 Subject: [PATCH 5/7] refactor: rename cluster-troubleshoot skill to investigate-alert Move SKILL.md and tools/ into a new investigate-alert/ subdirectory under cluster-troubleshoot/, which now serves as a parent for multiple troubleshooting skills. Narrow the skill scope to focus on investigating firing alerts rather than general cluster troubleshooting. Key changes: - Rename skill from cluster-troubleshoot to investigate-alert - Reorganize investigation protocol: add blast-radius scoping step, remove symptom-based discovery path, renumber remaining steps - Update terminology: "fix" to "remediation options" throughout - Simplify cluster-troubleshoot/README.md to describe the directory - Update eval system_prompt and test_cases to match new skill name Signed-off-by: Alberto Falossi Assisted-by: Claude Code:claude-opus-4-6 --- cluster-troubleshoot/README.md | 32 +-------- .../{ => investigate-alert}/SKILL.md | 65 +++++++++---------- .../tools/check-recent-changes.sh | 0 .../{ => investigate-alert}/tools/common.sh | 2 +- .../tools/diagnose-node.sh | 0 .../tools/diagnose-operator.sh | 0 .../tools/diagnose-pod.sh | 0 .../tools/diagnose-workload.sh | 0 .../tools/discover-metrics.sh | 0 .../tools/fetch-alert-rule.sh | 0 .../tools/get-events.sh | 0 .../tools/get-firing-alerts.sh | 0 .../tools/prometheus-query-range.sh | 0 .../tools/prometheus-query.sh | 0 .../tools/prometheus-setup.sh | 0 .../tools/query-alert.sh | 0 .../system_prompt.md | 4 +- .../test_cases.yaml | 36 +++++----- 18 files changed, 54 insertions(+), 85 deletions(-) rename cluster-troubleshoot/{ => investigate-alert}/SKILL.md (66%) rename cluster-troubleshoot/{ => investigate-alert}/tools/check-recent-changes.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/common.sh (98%) rename cluster-troubleshoot/{ => investigate-alert}/tools/diagnose-node.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/diagnose-operator.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/diagnose-pod.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/diagnose-workload.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/discover-metrics.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/fetch-alert-rule.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/get-events.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/get-firing-alerts.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/prometheus-query-range.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/prometheus-query.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/prometheus-setup.sh (100%) rename cluster-troubleshoot/{ => investigate-alert}/tools/query-alert.sh (100%) rename evals/skills/{cluster-troubleshoot => investigate-alert}/system_prompt.md (69%) rename evals/skills/{cluster-troubleshoot => investigate-alert}/test_cases.yaml (76%) diff --git a/cluster-troubleshoot/README.md b/cluster-troubleshoot/README.md index 13fa339..c5d37c7 100644 --- a/cluster-troubleshoot/README.md +++ b/cluster-troubleshoot/README.md @@ -1,31 +1,3 @@ -# Cluster Troubleshooting Skill +# Cluster Troubleshooting skills -Diagnose and troubleshoot OpenShift cluster issues: alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, and other cluster anomalies. - -## Structure - -``` -cluster-troubleshoot/ - SKILL.md # Skill definition (agent prompt) - tools/ # Diagnostic shell scripts (JSON output) -``` - -## Tools - -All scripts are in `tools/`, output JSON to stdout, and return structured error objects on failure. - -| Script | Purpose | -|--------|---------| -| `prometheus-setup.sh` | Get TOKEN and THANOS_URL for Prometheus access | -| `query-alert.sh` | Query firing instances of a named alert | -| `fetch-alert-rule.sh` | Get alert PromQL expression and annotations | -| `prometheus-query.sh` | Run an instant PromQL query | -| `prometheus-query-range.sh` | Run a range PromQL query | -| `discover-metrics.sh` | Search for available metric names | -| `get-firing-alerts.sh` | List all currently firing alerts | -| `diagnose-pod.sh` | Pod status, conditions, events, logs | -| `diagnose-node.sh` | Node conditions, capacity, taints | -| `diagnose-operator.sh` | ClusterOperator health | -| `diagnose-workload.sh` | Deployment/StatefulSet/DaemonSet status | -| `get-events.sh` | Events sorted by time | -| `check-recent-changes.sh` | Recent rollouts, image changes | +This directory contains skills which are designed to help agents diagnose and troubleshoot OpenShift cluster issues. diff --git a/cluster-troubleshoot/SKILL.md b/cluster-troubleshoot/investigate-alert/SKILL.md similarity index 66% rename from cluster-troubleshoot/SKILL.md rename to cluster-troubleshoot/investigate-alert/SKILL.md index 7f8faf5..8d017df 100644 --- a/cluster-troubleshoot/SKILL.md +++ b/cluster-troubleshoot/investigate-alert/SKILL.md @@ -1,7 +1,6 @@ --- -name: cluster-troubleshoot -description: Diagnose and troubleshoot OpenShift cluster issues. Use when the user reports alerts firing, pods crashing, deployments stuck, nodes not ready, operators degraded, HTTP errors, DNS failures, or any cluster anomaly. Not for cluster setup, configuration how-tos, or writing alerting rules. -allowed-tools: Bash +name: investigate-alert +description: Investigate one or more firing OpenShift alerts to determine root cause and recommend remediation options. Use when the user provides alert names or a set of firing alerts. Not for cluster setup, configuration how-tos, or writing alerting rules. --- # Environment @@ -16,14 +15,15 @@ allowed-tools: Bash - Verify every claim with evidence from command output. Provide exact resource names, namespaces, timestamps, and error messages. - If multiple causes exist, list them numbered with supporting evidence. - If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information. -- Stay focused on the reported issue — don't surface unrelated errors. -- The output remediation plan must address only the root cause of the specific alert or issue being analyzed. Never include secondary issues, unrelated findings, or general recommendations. +- Stay focused on the reported alerts; don't surface unrelated errors. +- The remediation options must address only the root cause of the alerts being analyzed. Never include secondary issues, unrelated findings, or general recommendations. - No URLs unless from command output or provided context. -- List actual resources before inspecting them — never guess pod names, use label selectors or owner references. +- List actual resources before inspecting them; never guess pod names, use label selectors or owner references. - Sample up to 3 representative pods per workload, not all. - All `tools/` scripts output JSON. Use `jq` for further filtering when needed. +- When running raw `oc` commands, use `-o json | jq` for structured data extraction. - Do not repeat the same command with the same arguments. -- Do not ask the user to run a command — gather the information yourself. +- Do not ask the user to run a command; gather the information yourself. - Be highly concise. Evidence-backed conclusions, no filler. # Tools @@ -51,60 +51,55 @@ The `tools/` directory contains diagnostic scripts. All output JSON to stdout an # Investigation Protocol -The protocol has two phases: **collect evidence first**, then **analyze**. Do not jump to conclusions or propose fixes until you have completed the collection phase. +The protocol has two phases: **collect evidence first**, then **analyze**. Do not jump to conclusions or propose remediation until you have completed the collection phase. ## Phase 1 — Collect evidence -### Step 1 — Set up and identify the problem signal +### Step 1 — Scope the blast radius + +Determine the scope of the alerts: is it one pod, one node, one namespace, or cluster-wide? This determines which layer to start from and which tools to prioritize. + +### Step 2 — Set up and query the alerts Configure Prometheus access: ```bash eval $(bash tools/prometheus-setup.sh) ``` -**If the user provides an alert name:** +For each alert provided: ```bash bash tools/query-alert.sh bash tools/fetch-alert-rule.sh ``` -If the alert is firing, extract the full label set (namespace, pod, node, service, severity, etc.). If it is NOT firing, still fetch the rule — the PromQL expression and labels contain diagnostic information. +Extract the full label set (namespace, pod, node, service, severity, etc.) from firing instances. Even if an alert is NOT firing, fetch the rule; the PromQL expression and labels contain diagnostic information. If the rule is found, check the annotations for `runbook_url`. If present, extract the URL and follow the runbook steps as part of the analysis. -Run the alert's PromQL expression to see current and recent values: +Run each alert's PromQL expression to see current and recent values: ```bash bash tools/prometheus-query.sh '' bash tools/prometheus-query-range.sh '' ``` -**If the user describes a symptom without an alert name** (e.g., "pods crashing in payments", "namespace X is broken"): -```bash -bash tools/get-firing-alerts.sh -``` -Filter the output for alerts matching the namespace or resources the user mentioned. If alerts are found, pick the most relevant one and fetch its rule as above. If no alerts are firing, use metric exploration: -```bash -bash tools/discover-metrics.sh -bash tools/discover-metrics.sh -``` -Query discovered metrics with `bash tools/prometheus-query.sh` to look for anomalies. - -### Step 2 — Collect workload state and logs from all workloads +### Step 3 — Collect workload state and logs -List all deployments, statefulsets, and daemonsets in the affected namespace. Then for **every** workload, collect its status and logs: +Using the namespaces and resources identified from the alert labels, list all deployments, statefulsets, and daemonsets in the affected namespaces. Then for **every** workload, collect its status and logs: ```bash bash tools/diagnose-workload.sh bash tools/diagnose-pod.sh --logs ``` This is mandatory. Do not skip workloads that appear healthy. A Running pod with no visible issues can still be the root cause if it is producing errors internally. -### Step 3 — Collect events and recent changes +### Step 4 — Collect events and recent changes ```bash bash tools/get-events.sh bash tools/check-recent-changes.sh ``` -### Step 4 — Collect firing alerts for correlation +The scripts cover rollouts and image pulls. Also check for: config/secret edits, HPA scaling events, operator upgrades, node drains, and MachineConfig updates using `oc rollout history`, `oc describe`, and events sorted by time. + +### Step 5 — Collect all firing alerts for correlation ```bash bash tools/get-firing-alerts.sh @@ -112,7 +107,7 @@ bash tools/get-firing-alerts.sh ## Phase 2 — Analyze and diagnose -### Step 5 — Correlate alerts +### Step 6 — Correlate alerts Group all firing alerts by shared labels to find the common root cause: - Alerts sharing the same **node** → likely a node-level issue (investigate the node first) @@ -122,7 +117,7 @@ Group all firing alerts by shared labels to find the common root cause: Prioritize investigation from infrastructure inward: node → operator → workload → pod. -### Step 6 — Trace causality chains +### Step 7 — Trace causality chains Use the evidence collected in phase 1 to trace the chain from symptom to root cause. If resource A fails because of B, investigate B. At each link, check status conditions and events. Common chains in OpenShift: - Pod pending → pod conditions (PodScheduled, Unschedulable) and events → node pressure, unschedulable nodes, resource quota, or PVC not bound @@ -131,15 +126,17 @@ Use the evidence collected in phase 1 to trace the chain from symptom to root ca - Operator degraded → `bash tools/diagnose-operator.sh ` (check Available, Degraded, Progressing with messages) → operand pod failing → node issue or config error - Node NotReady → node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and events → kubelet issues, certificate expired, MCO update stuck, or kernel panic -Use the range query data from step 1 and the recent changes from step 3 to identify **when** the problem started and what changed around that time. +Use the range query data from step 2 and the recent changes from step 4 to identify **when** the problem started and what changed around that time. + +After identifying a root cause, keep digging: collect exact names, versions, and labels, and check for additional contributing factors before concluding. -### Step 7 — Recommend a fix +### Step 8 — Recommend remediation options -Based on the collected evidence, propose a remediation. Provide the exact commands to run. When multiple options exist, list them from least to most disruptive and note whether each action is reversible. If the root cause is unclear, state what is known, suggest mitigations to reduce impact, and identify what additional data would narrow the diagnosis. +Based on the collected evidence, propose remediation options. Provide the exact commands to run. When multiple options exist, list them from least to most disruptive and note whether each action is reversible. If the root cause is unclear, state what is known, suggest mitigations to reduce impact, and identify what additional data would narrow the diagnosis. -**RBAC requirements for remediation:** the agent's ServiceAccount has **read-only** permissions (`cluster-reader` + `cluster-monitoring-view`). Any proposed remediation command (patch, delete, scale, rollout restart, etc.) requires additional RBAC that the agent does not have. For every proposed fix, you **must** include a complete RBAC section listing: +**RBAC requirements for remediation:** the agent's ServiceAccount has **read-only** permissions (`cluster-reader` + `cluster-monitoring-view`). Any proposed remediation command (patch, delete, scale, rollout restart, etc.) requires additional RBAC that the agent does not have. For every proposed option, you **must** include a complete RBAC section listing: - The exact API group, resource, and verbs required (e.g., `apps` / `deployments` / `patch`) - The namespace scope (specific namespace or cluster-wide) - A ready-to-apply Role or ClusterRole YAML snippet the admin can use to grant the permissions -This is critical — without this information the admin cannot safely execute the proposed fix. +This is critical; without this information the admin cannot safely execute the proposed remediation. diff --git a/cluster-troubleshoot/tools/check-recent-changes.sh b/cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh similarity index 100% rename from cluster-troubleshoot/tools/check-recent-changes.sh rename to cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh diff --git a/cluster-troubleshoot/tools/common.sh b/cluster-troubleshoot/investigate-alert/tools/common.sh similarity index 98% rename from cluster-troubleshoot/tools/common.sh rename to cluster-troubleshoot/investigate-alert/tools/common.sh index 8902df8..91c5a4b 100755 --- a/cluster-troubleshoot/tools/common.sh +++ b/cluster-troubleshoot/investigate-alert/tools/common.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Shared functions for cluster-troubleshoot tools. +# Shared functions for investigate-alert tools. # Source this file: . "$(dirname "$0")/common.sh" set -euo pipefail diff --git a/cluster-troubleshoot/tools/diagnose-node.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-node.sh similarity index 100% rename from cluster-troubleshoot/tools/diagnose-node.sh rename to cluster-troubleshoot/investigate-alert/tools/diagnose-node.sh diff --git a/cluster-troubleshoot/tools/diagnose-operator.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-operator.sh similarity index 100% rename from cluster-troubleshoot/tools/diagnose-operator.sh rename to cluster-troubleshoot/investigate-alert/tools/diagnose-operator.sh diff --git a/cluster-troubleshoot/tools/diagnose-pod.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-pod.sh similarity index 100% rename from cluster-troubleshoot/tools/diagnose-pod.sh rename to cluster-troubleshoot/investigate-alert/tools/diagnose-pod.sh diff --git a/cluster-troubleshoot/tools/diagnose-workload.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh similarity index 100% rename from cluster-troubleshoot/tools/diagnose-workload.sh rename to cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh diff --git a/cluster-troubleshoot/tools/discover-metrics.sh b/cluster-troubleshoot/investigate-alert/tools/discover-metrics.sh similarity index 100% rename from cluster-troubleshoot/tools/discover-metrics.sh rename to cluster-troubleshoot/investigate-alert/tools/discover-metrics.sh diff --git a/cluster-troubleshoot/tools/fetch-alert-rule.sh b/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh similarity index 100% rename from cluster-troubleshoot/tools/fetch-alert-rule.sh rename to cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh diff --git a/cluster-troubleshoot/tools/get-events.sh b/cluster-troubleshoot/investigate-alert/tools/get-events.sh similarity index 100% rename from cluster-troubleshoot/tools/get-events.sh rename to cluster-troubleshoot/investigate-alert/tools/get-events.sh diff --git a/cluster-troubleshoot/tools/get-firing-alerts.sh b/cluster-troubleshoot/investigate-alert/tools/get-firing-alerts.sh similarity index 100% rename from cluster-troubleshoot/tools/get-firing-alerts.sh rename to cluster-troubleshoot/investigate-alert/tools/get-firing-alerts.sh diff --git a/cluster-troubleshoot/tools/prometheus-query-range.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-query-range.sh similarity index 100% rename from cluster-troubleshoot/tools/prometheus-query-range.sh rename to cluster-troubleshoot/investigate-alert/tools/prometheus-query-range.sh diff --git a/cluster-troubleshoot/tools/prometheus-query.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-query.sh similarity index 100% rename from cluster-troubleshoot/tools/prometheus-query.sh rename to cluster-troubleshoot/investigate-alert/tools/prometheus-query.sh diff --git a/cluster-troubleshoot/tools/prometheus-setup.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh similarity index 100% rename from cluster-troubleshoot/tools/prometheus-setup.sh rename to cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh diff --git a/cluster-troubleshoot/tools/query-alert.sh b/cluster-troubleshoot/investigate-alert/tools/query-alert.sh similarity index 100% rename from cluster-troubleshoot/tools/query-alert.sh rename to cluster-troubleshoot/investigate-alert/tools/query-alert.sh diff --git a/evals/skills/cluster-troubleshoot/system_prompt.md b/evals/skills/investigate-alert/system_prompt.md similarity index 69% rename from evals/skills/cluster-troubleshoot/system_prompt.md rename to evals/skills/investigate-alert/system_prompt.md index 41d5c09..2090cd0 100644 --- a/evals/skills/cluster-troubleshoot/system_prompt.md +++ b/evals/skills/investigate-alert/system_prompt.md @@ -1,4 +1,4 @@ -You are a cluster diagnostics assistant with access to the 'cluster-troubleshoot' skill. This skill provides shell scripts under tools/ for diagnosing OpenShift cluster issues: pod crashes, node failures, operator degradation, firing alerts, and other anomalies. +You are an alert investigation assistant with access to the 'investigate-alert' skill. This skill provides shell scripts under tools/ for investigating firing OpenShift alerts: determining root cause and recommending remediation options. Available diagnostic scripts (all under tools/): - prometheus-setup.sh — Set TOKEN and THANOS_URL for Prometheus access @@ -15,4 +15,4 @@ Available diagnostic scripts (all under tools/): - get-events.sh [namespace] [minutes] — Events sorted by time - check-recent-changes.sh [namespace] [minutes] — Recent rollouts, image pulls -The investigation protocol has two phases: collect evidence first (set up Prometheus, identify problem signal, collect workload state and logs, collect events and recent changes, collect firing alerts), then analyze (correlate alerts, trace causality chains, recommend a fix). +The investigation protocol has two phases: collect evidence first (scope blast radius, set up Prometheus, query alerts, collect workload state and logs, collect events and recent changes, collect all firing alerts), then analyze (correlate alerts, trace causality chains, recommend remediation options). diff --git a/evals/skills/cluster-troubleshoot/test_cases.yaml b/evals/skills/investigate-alert/test_cases.yaml similarity index 76% rename from evals/skills/cluster-troubleshoot/test_cases.yaml rename to evals/skills/investigate-alert/test_cases.yaml index 3182a81..d2eb1e3 100644 --- a/evals/skills/cluster-troubleshoot/test_cases.yaml +++ b/evals/skills/investigate-alert/test_cases.yaml @@ -1,4 +1,4 @@ -# cluster-troubleshoot skill eval test cases +# investigate-alert skill eval test cases # # Static tests verify tool-selection knowledge — the agent must pick the # correct diagnostic script for a given scenario. @@ -9,98 +9,98 @@ # --- Static tool-selection tests --- - name: tool_for_crashing_pod - query: "A pod named web-frontend-7b9d4 is crash-looping in namespace production. Which cluster-troubleshoot diagnostic script should I run first to inspect its status, conditions, and events?" + query: "A pod named web-frontend-7b9d4 is crash-looping in namespace production. Which investigate-alert diagnostic script should I run first to inspect its status, conditions, and events?" schema: type: object properties: script: type: string enum: ["diagnose-pod.sh", "diagnose-workload.sh", "diagnose-node.sh", "get-events.sh", "get-firing-alerts.sh"] - description: "The diagnostic script from the cluster-troubleshoot skill to run first. Pick the most specific one." + description: "The diagnostic script from the investigate-alert skill to run first. Pick the most specific one." required: [script] expected: script: "diagnose-pod.sh" - name: tool_for_firing_alert - query: "An alert called KubePodCrashLooping is firing. Which cluster-troubleshoot diagnostic script fetches the alert's PromQL rule definition, thresholds, and annotations?" + query: "An alert called KubePodCrashLooping is firing. Which investigate-alert diagnostic script fetches the alert's PromQL rule definition, thresholds, and annotations?" schema: type: object properties: script: type: string enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"] - description: "The script that retrieves the alert rule definition. Use the 'cluster-troubleshoot' skill to find this." + description: "The script that retrieves the alert rule definition. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "fetch-alert-rule.sh" - name: tool_for_node_issue - query: "A node is showing NotReady status. Which cluster-troubleshoot diagnostic script provides node conditions, capacity, taints, and kubelet info?" + query: "A node is showing NotReady status. Which investigate-alert diagnostic script provides node conditions, capacity, taints, and kubelet info?" schema: type: object properties: script: type: string enum: ["diagnose-node.sh", "diagnose-pod.sh", "diagnose-operator.sh", "get-events.sh", "check-recent-changes.sh"] - description: "The script for node diagnostics. Use the 'cluster-troubleshoot' skill to find this." + description: "The script for node diagnostics. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "diagnose-node.sh" - name: tool_for_operator_degraded - query: "The authentication ClusterOperator is showing Degraded=True. Which cluster-troubleshoot diagnostic script checks ClusterOperator health, conditions, and versions?" + query: "The authentication ClusterOperator is showing Degraded=True. Which investigate-alert diagnostic script checks ClusterOperator health, conditions, and versions?" schema: type: object properties: script: type: string enum: ["diagnose-operator.sh", "diagnose-workload.sh", "diagnose-pod.sh", "get-firing-alerts.sh", "check-recent-changes.sh"] - description: "The script for ClusterOperator diagnostics. Use the 'cluster-troubleshoot' skill to find this." + description: "The script for ClusterOperator diagnostics. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "diagnose-operator.sh" - name: tool_for_prometheus_setup - query: "Before running any Prometheus queries with the cluster-troubleshoot skill, what script must be run first to configure access credentials?" + query: "Before running any Prometheus queries with the investigate-alert skill, what script must be run first to configure access credentials?" schema: type: object properties: script: type: string enum: ["prometheus-setup.sh", "prometheus-query.sh", "discover-metrics.sh", "get-firing-alerts.sh"] - description: "The script that sets up Prometheus credentials. Use the 'cluster-troubleshoot' skill to find this." + description: "The script that sets up Prometheus credentials. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "prometheus-setup.sh" - name: tool_for_recent_changes - query: "Something changed recently and broke our namespace. Which cluster-troubleshoot diagnostic script detects recent rollouts, image pulls, and scaling events?" + query: "Something changed recently and broke our namespace. Which investigate-alert diagnostic script detects recent rollouts, image pulls, and scaling events?" schema: type: object properties: script: type: string enum: ["check-recent-changes.sh", "get-events.sh", "diagnose-workload.sh", "get-firing-alerts.sh", "diagnose-pod.sh"] - description: "The script for detecting recent cluster changes. Use the 'cluster-troubleshoot' skill to find this." + description: "The script for detecting recent cluster changes. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "check-recent-changes.sh" - name: diagnose_pod_log_flag - query: "When using the cluster-troubleshoot skill's diagnose-pod.sh, what command-line flag do you pass to include container logs in the output?" + query: "When using the investigate-alert skill's diagnose-pod.sh, what command-line flag do you pass to include container logs in the output?" schema: type: object properties: flag: type: string enum: ["--logs", "--include-logs", "-l", "--verbose", "--tail"] - description: "The flag for including logs. Use the 'cluster-troubleshoot' skill to find this." + description: "The flag for including logs. Use the 'investigate-alert' skill to find this." required: [flag] expected: flag: "--logs" - name: first_investigation_step - query: "According to the cluster-troubleshoot investigation protocol, what is the very first command you must run in Step 1 before anything else?" + query: "According to the investigate-alert investigation protocol, what is the first command you must run in Step 2 to set up Prometheus access?" schema: type: object properties: @@ -111,7 +111,7 @@ - "bash tools/get-firing-alerts.sh" - "bash tools/diagnose-pod.sh" - "bash tools/get-events.sh" - description: "The first command in the investigation protocol. Use the 'cluster-troubleshoot' skill to find this." + description: "The first command in the investigation protocol. Use the 'investigate-alert' skill to find this." required: [command] expected: command: "eval $(bash tools/prometheus-setup.sh)" @@ -129,7 +129,7 @@ root_cause: type: string enum: ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff", "Evicted", "NodeNotReady"] - description: "The underlying cause of the container's termination. Use the 'cluster-troubleshoot' skill to diagnose." + description: "The underlying cause of the container's termination. Use the 'investigate-alert' skill to diagnose." pod_name: type: string description: "The name of the affected pod" From 16e96fb4f7fc2a9c26b32b41d87c0f4209f51d23 Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Wed, 22 Jul 2026 10:15:05 +0200 Subject: [PATCH 6/7] fix: harden investigate-alert tool input handling - Quote exported values with %q in prometheus-setup.sh - Use jq -n for JSON construction in fetch-alert-rule.sh - Validate alert name format in query-alert.sh - Add default case for unsupported kinds in diagnose-workload.sh - Amend "All output JSON" claim to note prometheus-setup.sh exception Signed-off-by: Alberto Falossi Assisted-by: Claude Code:claude-opus-4-6 --- cluster-troubleshoot/investigate-alert/SKILL.md | 4 ++-- .../investigate-alert/tools/diagnose-workload.sh | 4 ++++ .../investigate-alert/tools/fetch-alert-rule.sh | 2 +- .../investigate-alert/tools/prometheus-setup.sh | 4 ++-- cluster-troubleshoot/investigate-alert/tools/query-alert.sh | 4 ++++ 5 files changed, 13 insertions(+), 5 deletions(-) diff --git a/cluster-troubleshoot/investigate-alert/SKILL.md b/cluster-troubleshoot/investigate-alert/SKILL.md index 8d017df..0e76104 100644 --- a/cluster-troubleshoot/investigate-alert/SKILL.md +++ b/cluster-troubleshoot/investigate-alert/SKILL.md @@ -20,7 +20,7 @@ description: Investigate one or more firing OpenShift alerts to determine root c - No URLs unless from command output or provided context. - List actual resources before inspecting them; never guess pod names, use label selectors or owner references. - Sample up to 3 representative pods per workload, not all. -- All `tools/` scripts output JSON. Use `jq` for further filtering when needed. +- All `tools/` scripts output JSON except `prometheus-setup.sh` (shell export commands for `eval`). Use `jq` for further filtering when needed. - When running raw `oc` commands, use `-o json | jq` for structured data extraction. - Do not repeat the same command with the same arguments. - Do not ask the user to run a command; gather the information yourself. @@ -28,7 +28,7 @@ description: Investigate one or more firing OpenShift alerts to determine root c # Tools -The `tools/` directory contains diagnostic scripts. All output JSON to stdout and return structured error objects on failure. Run `eval $(bash tools/prometheus-setup.sh)` once per session before using Prometheus tools. +The `tools/` directory contains diagnostic scripts. All output JSON to stdout and return structured error objects on failure, except `prometheus-setup.sh` which prints shell export commands for `eval`. Run `eval $(bash tools/prometheus-setup.sh)` once per session before using Prometheus tools. ## Prometheus diff --git a/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh index 48baedc..2f5424d 100755 --- a/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh +++ b/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh @@ -99,4 +99,8 @@ case "$KIND" in rollout_history: $hist }' ;; + *) + error_json "UNSUPPORTED_KIND" "Unsupported workload kind: $KIND" \ + "Supported kinds: deployment, statefulset, daemonset" + ;; esac diff --git a/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh b/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh index 9e431e0..8e51b7a 100755 --- a/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh +++ b/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh @@ -18,7 +18,7 @@ RULE=$(echo "$RESPONSE" | jq --arg name "$ALERT_NAME" \ '[.data.groups[].rules[] | select(.name==$name and .type=="alerting")] | first // empty') if [[ -z "$RULE" || "$RULE" == "null" ]]; then - printf '{"alert_name":"%s","found":false}\n' "$ALERT_NAME" + jq -n --arg name "$ALERT_NAME" '{alert_name: $name, found: false}' else echo "$RULE" | jq --arg name "$ALERT_NAME" '{ alert_name: $name, diff --git a/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh index 17a74c9..e9badb4 100755 --- a/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh +++ b/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh @@ -23,5 +23,5 @@ if [[ -z "$THANOS_URL" ]]; then "Check: oc get route thanos-querier -n openshift-monitoring" fi -printf 'export TOKEN=%s\n' "$TOKEN" -printf 'export THANOS_URL=%s\n' "$THANOS_URL" +printf 'export TOKEN=%q\n' "$TOKEN" +printf 'export THANOS_URL=%q\n' "$THANOS_URL" diff --git a/cluster-troubleshoot/investigate-alert/tools/query-alert.sh b/cluster-troubleshoot/investigate-alert/tools/query-alert.sh index 675e292..8256cfe 100755 --- a/cluster-troubleshoot/investigate-alert/tools/query-alert.sh +++ b/cluster-troubleshoot/investigate-alert/tools/query-alert.sh @@ -12,6 +12,10 @@ require_command python3 require_arg "${1:-}" "alert name" "Usage: query-alert.sh " ALERT_NAME="$1" +if [[ ! "$ALERT_NAME" =~ ^[a-zA-Z_:][a-zA-Z0-9_:]*$ ]]; then + error_json "INVALID_ARG" "Invalid alert name: '$ALERT_NAME'" \ + "Alert names must match ^[a-zA-Z_:][a-zA-Z0-9_:]*$" +fi PROMQL="ALERTS{alertname=\"${ALERT_NAME}\"}" QUERY=$(urlencode "$PROMQL") From a33d54b022b5791f5a4c97dd1bece9217f9fb7a0 Mon Sep 17 00:00:00 2001 From: Alberto Falossi Date: Wed, 22 Jul 2026 10:35:28 +0200 Subject: [PATCH 7/7] refactor: reframe investigate-alert evals as alert-driven Rewrite all test cases to start from a firing alert as the trigger point, matching the skill's identity shift from generic cluster troubleshooting to alert-specific investigation. - Tool-selection tests now begin with "you are investigating alert X" - Mock-based tests include alert context in the scenario - Add step3_mandatory_logs test for protocol evidence requirements - Add fetch_alert_rule test (query-alert vs fetch-alert-rule distinction) - Drop diagnose_pod_log_flag (flag trivia, not investigation-relevant) Signed-off-by: Alberto Falossi Assisted-by: Claude Code:claude-opus-4-6 --- .../skills/investigate-alert/test_cases.yaml | 130 ++++++++++-------- 1 file changed, 73 insertions(+), 57 deletions(-) diff --git a/evals/skills/investigate-alert/test_cases.yaml b/evals/skills/investigate-alert/test_cases.yaml index d2eb1e3..9a5375e 100644 --- a/evals/skills/investigate-alert/test_cases.yaml +++ b/evals/skills/investigate-alert/test_cases.yaml @@ -1,80 +1,83 @@ # investigate-alert skill eval test cases # -# Static tests verify tool-selection knowledge — the agent must pick the -# correct diagnostic script for a given scenario. +# All tests are framed around the skill's core workflow: an alert fires, +# and the agent follows the investigation protocol to determine root cause. # -# Mock-based tests embed canned tool output in the query and verify the -# agent interprets the diagnostic data correctly. +# Static tests verify tool-selection and protocol knowledge — given a step +# in the investigation, the agent must pick the correct diagnostic script. +# +# Mock-based tests embed canned tool output and verify the agent +# interprets diagnostic data correctly in an alert context. # --- Static tool-selection tests --- -- name: tool_for_crashing_pod - query: "A pod named web-frontend-7b9d4 is crash-looping in namespace production. Which investigate-alert diagnostic script should I run first to inspect its status, conditions, and events?" +- name: query_firing_alert + query: "Alert KubePodCrashLooping is firing. Which investigate-alert diagnostic script should you run to check how many instances are firing and get their full label sets?" schema: type: object properties: script: type: string - enum: ["diagnose-pod.sh", "diagnose-workload.sh", "diagnose-node.sh", "get-events.sh", "get-firing-alerts.sh"] - description: "The diagnostic script from the investigate-alert skill to run first. Pick the most specific one." + enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"] + description: "The script that queries firing instances of a named alert. Use the 'investigate-alert' skill to find this." required: [script] expected: - script: "diagnose-pod.sh" + script: "query-alert.sh" -- name: tool_for_firing_alert - query: "An alert called KubePodCrashLooping is firing. Which investigate-alert diagnostic script fetches the alert's PromQL rule definition, thresholds, and annotations?" +- name: fetch_alert_rule + query: "You are investigating alert KubeDeploymentReplicasMismatch. Which investigate-alert diagnostic script retrieves its PromQL expression, thresholds, and annotations?" schema: type: object properties: script: type: string enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"] - description: "The script that retrieves the alert rule definition. Use the 'investigate-alert' skill to find this." + description: "The script that fetches the alert rule definition. Use the 'investigate-alert' skill to find this." required: [script] expected: script: "fetch-alert-rule.sh" -- name: tool_for_node_issue - query: "A node is showing NotReady status. Which investigate-alert diagnostic script provides node conditions, capacity, taints, and kubelet info?" +- name: diagnose_pod_from_alert + query: "You are investigating alert KubePodCrashLooping. The alert labels show pod web-frontend-7b9d4 in namespace production. Which investigate-alert diagnostic script should you use to inspect this pod's status, conditions, and events?" schema: type: object properties: script: type: string - enum: ["diagnose-node.sh", "diagnose-pod.sh", "diagnose-operator.sh", "get-events.sh", "check-recent-changes.sh"] - description: "The script for node diagnostics. Use the 'investigate-alert' skill to find this." + enum: ["diagnose-pod.sh", "diagnose-workload.sh", "diagnose-node.sh", "get-events.sh", "get-firing-alerts.sh"] + description: "The script for pod-level diagnostics. Use the 'investigate-alert' skill to find this." required: [script] expected: - script: "diagnose-node.sh" + script: "diagnose-pod.sh" -- name: tool_for_operator_degraded - query: "The authentication ClusterOperator is showing Degraded=True. Which investigate-alert diagnostic script checks ClusterOperator health, conditions, and versions?" +- name: diagnose_node_from_alert + query: "You are investigating alert KubeNodeNotReady. The alert labels point to node worker-3. Which investigate-alert diagnostic script provides node conditions, capacity, taints, and kubelet info?" schema: type: object properties: script: type: string - enum: ["diagnose-operator.sh", "diagnose-workload.sh", "diagnose-pod.sh", "get-firing-alerts.sh", "check-recent-changes.sh"] - description: "The script for ClusterOperator diagnostics. Use the 'investigate-alert' skill to find this." + enum: ["diagnose-node.sh", "diagnose-pod.sh", "diagnose-operator.sh", "get-events.sh", "check-recent-changes.sh"] + description: "The script for node diagnostics. Use the 'investigate-alert' skill to find this." required: [script] expected: - script: "diagnose-operator.sh" + script: "diagnose-node.sh" -- name: tool_for_prometheus_setup - query: "Before running any Prometheus queries with the investigate-alert skill, what script must be run first to configure access credentials?" +- name: diagnose_operator_from_alert + query: "You are investigating alert ClusterOperatorDegraded. The alert labels show operator name 'authentication'. Which investigate-alert diagnostic script checks its health, conditions, and versions?" schema: type: object properties: script: type: string - enum: ["prometheus-setup.sh", "prometheus-query.sh", "discover-metrics.sh", "get-firing-alerts.sh"] - description: "The script that sets up Prometheus credentials. Use the 'investigate-alert' skill to find this." + enum: ["diagnose-operator.sh", "diagnose-workload.sh", "diagnose-pod.sh", "get-firing-alerts.sh", "check-recent-changes.sh"] + description: "The script for ClusterOperator diagnostics. Use the 'investigate-alert' skill to find this." required: [script] expected: - script: "prometheus-setup.sh" + script: "diagnose-operator.sh" -- name: tool_for_recent_changes - query: "Something changed recently and broke our namespace. Which investigate-alert diagnostic script detects recent rollouts, image pulls, and scaling events?" +- name: check_changes_during_alert + query: "You are investigating alert KubeDeploymentReplicasMismatch in namespace checkout. You need to check what changed recently that may have triggered this alert. Which investigate-alert diagnostic script detects recent rollouts, image pulls, and scaling events?" schema: type: object properties: @@ -86,18 +89,18 @@ expected: script: "check-recent-changes.sh" -- name: diagnose_pod_log_flag - query: "When using the investigate-alert skill's diagnose-pod.sh, what command-line flag do you pass to include container logs in the output?" +- name: prometheus_setup + query: "Before querying any alerts with the investigate-alert skill's Prometheus tools, what script must be run first to configure access credentials?" schema: type: object properties: - flag: + script: type: string - enum: ["--logs", "--include-logs", "-l", "--verbose", "--tail"] - description: "The flag for including logs. Use the 'investigate-alert' skill to find this." - required: [flag] + enum: ["prometheus-setup.sh", "prometheus-query.sh", "discover-metrics.sh", "get-firing-alerts.sh"] + description: "The script that sets up Prometheus credentials. Use the 'investigate-alert' skill to find this." + required: [script] expected: - flag: "--logs" + script: "prometheus-setup.sh" - name: first_investigation_step query: "According to the investigate-alert investigation protocol, what is the first command you must run in Step 2 to set up Prometheus access?" @@ -116,35 +119,24 @@ expected: command: "eval $(bash tools/prometheus-setup.sh)" -# --- Mock-based diagnostic reasoning tests --- - -- name: interpret_pod_oomkilled - query: | - You ran diagnose-pod.sh and got this output: - {"pods":[{"name":"api-server-6f8b9c-xk2lm","namespace":"payments","phase":"Running","conditions":[{"type":"Ready","status":"False"},{"type":"ContainersReady","status":"False"}],"container_statuses":[{"name":"api","state":{"waiting":{"reason":"CrashLoopBackOff"}},"lastState":{"terminated":{"reason":"OOMKilled","exitCode":137,"finishedAt":"2026-07-17T08:15:00Z"}},"restartCount":12,"ready":false}],"node_name":"worker-2","restart_count":12,"events":[{"type":"Warning","reason":"BackOff","message":"Back-off restarting failed container api in pod api-server-6f8b9c-xk2lm_payments"}]}]} - What is the root cause of this pod's failure? +- name: step3_mandatory_logs + query: "You are investigating alert KubeDeploymentReplicasMismatch. You have already run diagnose-workload.sh for the affected deployment. According to the investigation protocol Step 3, what must you also run for every workload to collect container logs, even if the workload appears healthy?" schema: type: object properties: - root_cause: - type: string - enum: ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff", "Evicted", "NodeNotReady"] - description: "The underlying cause of the container's termination. Use the 'investigate-alert' skill to diagnose." - pod_name: + tool: type: string - description: "The name of the affected pod" - restart_count: - type: integer - description: "Number of container restarts" - required: [root_cause, pod_name, restart_count] + enum: ["diagnose-pod.sh --logs", "get-events.sh", "check-recent-changes.sh", "prometheus-query.sh"] + description: "The additional evidence to collect per workload in Step 3. Use the 'investigate-alert' skill to find this." + required: [tool] expected: - root_cause: "OOMKilled" - pod_name: "api-server-6f8b9c-xk2lm" - restart_count: 12 + tool: "diagnose-pod.sh --logs" + +# --- Mock-based diagnostic reasoning tests --- - name: interpret_firing_alert query: | - You ran query-alert.sh KubeDeploymentReplicasMismatch and got: + You are investigating alert KubeDeploymentReplicasMismatch. You ran query-alert.sh and got: {"alert_name":"KubeDeploymentReplicasMismatch","firing":true,"instance_count":2,"instances":[{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"payment-gateway","severity":"warning"},"value":[1721203200,"1"]},{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"order-service","severity":"warning"},"value":[1721203200,"1"]}]} Based on this output: is the alert firing, how many instances are affected, and in which namespace? schema: @@ -165,3 +157,27 @@ is_firing: true instance_count: 2 namespace: "checkout" + +- name: interpret_pod_oomkilled + query: | + You are investigating alert KubePodCrashLooping. The alert labels pointed to pod api-server-6f8b9c-xk2lm in namespace payments. You ran diagnose-pod.sh and got this output: + {"pods":[{"name":"api-server-6f8b9c-xk2lm","namespace":"payments","phase":"Running","conditions":[{"type":"Ready","status":"False"},{"type":"ContainersReady","status":"False"}],"container_statuses":[{"name":"api","state":{"waiting":{"reason":"CrashLoopBackOff"}},"lastState":{"terminated":{"reason":"OOMKilled","exitCode":137,"finishedAt":"2026-07-17T08:15:00Z"}},"restartCount":12,"ready":false}],"node_name":"worker-2","restart_count":12,"events":[{"type":"Warning","reason":"BackOff","message":"Back-off restarting failed container api in pod api-server-6f8b9c-xk2lm_payments"}]}]} + What is the root cause of this pod's failure? + schema: + type: object + properties: + root_cause: + type: string + enum: ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff", "Evicted", "NodeNotReady"] + description: "The underlying cause of the container's termination." + pod_name: + type: string + description: "The name of the affected pod" + restart_count: + type: integer + description: "Number of container restarts" + required: [root_cause, pod_name, restart_count] + expected: + root_cause: "OOMKilled" + pod_name: "api-server-6f8b9c-xk2lm" + restart_count: 12