feat: add cluster-troubleshoot skill

cluster-troubleshoot/OWNERS

@@ -0,0 +1,16 @@
+# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md
+
+approvers:
+  - falox
+  - tremes
+  - rioloc
+  - shwetaap
+  - iNecas
+reviewers:
+  - falox
+  - tremes
+  - rioloc
+  - shwetaap
+  - iNecas
+
+component: "Cluster Troubleshooting Lightspeed Skills"

cluster-troubleshoot/README.md

@@ -0,0 +1,3 @@
+# Cluster Troubleshooting skills
+
+This directory contains skills which are designed to help agents diagnose and troubleshoot OpenShift cluster issues.

cluster-troubleshoot/investigate-alert/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: investigate-alert
+description: Investigate one or more firing OpenShift alerts to determine root cause and recommend remediation options. Use when the user provides alert names or a set of firing alerts. Not for cluster setup, configuration how-tos, or writing alerting rules.
+---
+
+# Environment
+
+- Platform: OpenShift Container Platform (OCP). Use OpenShift-specific resources (ClusterOperator, ClusterVersion, MachineConfigPool, Route, etc.) alongside standard Kubernetes ones.
+- You run commands against the live cluster.
+- Available CLIs: oc, kubectl, jq, wget, openssl, skopeo, python3, dig, nslookup, ip, ss, tcpdump, strace
+- Writable directories: /home/agent, /tmp/agent-workspace. The root filesystem is read-only.
+
+# Rules
+
+- Verify every claim with evidence from command output. Provide exact resource names, namespaces, timestamps, and error messages.
+- If multiple causes exist, list them numbered with supporting evidence.
+- If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information.
+- Stay focused on the reported alerts; don't surface unrelated errors.
+- The remediation options must address only the root cause of the alerts being analyzed. Never include secondary issues, unrelated findings, or general recommendations.
+- No URLs unless from command output or provided context.
+- List actual resources before inspecting them; never guess pod names, use label selectors or owner references.
+- Sample up to 3 representative pods per workload, not all.
+- All `tools/` scripts output JSON except `prometheus-setup.sh` (shell export commands for `eval`). Use `jq` for further filtering when needed.
+- When running raw `oc` commands, use `-o json | jq` for structured data extraction.
+- Do not repeat the same command with the same arguments.
+- Do not ask the user to run a command; gather the information yourself.
+- Be highly concise. Evidence-backed conclusions, no filler.
+
+# Tools
+
+The `tools/` directory contains diagnostic scripts. All output JSON to stdout and return structured error objects on failure, except `prometheus-setup.sh` which prints shell export commands for `eval`. Run `eval $(bash tools/prometheus-setup.sh)` once per session before using Prometheus tools.
+
+## Prometheus
+
+- `bash tools/prometheus-setup.sh` — Set TOKEN and THANOS_URL
+- `bash tools/query-alert.sh <alert_name>` — Firing instances of a named alert with full label sets
+- `bash tools/fetch-alert-rule.sh <alert_name>` — Alert PromQL expression, thresholds, annotations
+- `bash tools/prometheus-query.sh '<promql>'` — Instant PromQL query
+- `bash tools/prometheus-query-range.sh '<promql>' [duration] [step]` — Range query (default: 1h, 60s step)
+- `bash tools/discover-metrics.sh <pattern>` — Search metric names by pattern
+- `bash tools/get-firing-alerts.sh [filter]` — All firing alerts, optionally filtered by name
+
+## Cluster
+
+- `bash tools/diagnose-pod.sh <name-or-selector> [namespace] [--logs]` — Pod status, conditions, events, logs. Selectors (containing `=`) sample up to 3 pods.
+- `bash tools/diagnose-node.sh [node_name]` — Node conditions, capacity, taints. Without a name: all nodes summary.
+- `bash tools/diagnose-operator.sh [operator_name]` — ClusterOperator health. Without a name: all operators summary.
+- `bash tools/diagnose-workload.sh <name> [namespace] [kind]` — Deployment/StatefulSet/DaemonSet status, replicas, rollout history. Auto-detects kind.
+- `bash tools/get-events.sh [namespace] [minutes]` — Events sorted by time, last N minutes (default: 60). Use `--all` for cluster-wide.
+- `bash tools/check-recent-changes.sh [namespace] [minutes]` — Recent rollouts, image pulls, active rollouts.
+
+# Investigation Protocol
+
+The protocol has two phases: **collect evidence first**, then **analyze**. Do not jump to conclusions or propose remediation until you have completed the collection phase.
+
+## Phase 1 — Collect evidence
+
+### Step 1 — Scope the blast radius
+
+Determine the scope of the alerts: is it one pod, one node, one namespace, or cluster-wide? This determines which layer to start from and which tools to prioritize.
+
+### Step 2 — Set up and query the alerts
+
+Configure Prometheus access:
+```bash
+eval $(bash tools/prometheus-setup.sh)
+```
+
+For each alert provided:
+```bash
+bash tools/query-alert.sh <ALERT_NAME>
+bash tools/fetch-alert-rule.sh <ALERT_NAME>
+```
+Extract the full label set (namespace, pod, node, service, severity, etc.) from firing instances. Even if an alert is NOT firing, fetch the rule; the PromQL expression and labels contain diagnostic information.
+
+If the rule is found, check the annotations for `runbook_url`. If present, extract the URL and follow the runbook steps as part of the analysis.
+
+Run each alert's PromQL expression to see current and recent values:
+```bash
+bash tools/prometheus-query.sh '<expr>'
+bash tools/prometheus-query-range.sh '<expr>'
+```
+
+### Step 3 — Collect workload state and logs
+
+Using the namespaces and resources identified from the alert labels, list all deployments, statefulsets, and daemonsets in the affected namespaces. Then for **every** workload, collect its status and logs:
+```bash
+bash tools/diagnose-workload.sh <name> <namespace>
+bash tools/diagnose-pod.sh <name-or-selector> <namespace> --logs
+```
+This is mandatory. Do not skip workloads that appear healthy. A Running pod with no visible issues can still be the root cause if it is producing errors internally.
+
+### Step 4 — Collect events and recent changes
+
+```bash
+bash tools/get-events.sh <namespace>
+bash tools/check-recent-changes.sh <namespace>
+```
+
+The scripts cover rollouts and image pulls. Also check for: config/secret edits, HPA scaling events, operator upgrades, node drains, and MachineConfig updates using `oc rollout history`, `oc describe`, and events sorted by time.
+
+### Step 5 — Collect all firing alerts for correlation
+
+```bash
+bash tools/get-firing-alerts.sh
+```
+
+## Phase 2 — Analyze and diagnose
+
+### Step 6 — Correlate alerts
+
+Group all firing alerts by shared labels to find the common root cause:
+- Alerts sharing the same **node** → likely a node-level issue (investigate the node first)
+- Alerts sharing the same **namespace** but on different nodes → likely an application or config issue
+- Alerts sharing the same **job** or **service** → dependency or networking issue
+- Alerts across multiple namespaces and nodes → cluster-wide issue (check operators, control plane, networking)
+
+Prioritize investigation from infrastructure inward: node → operator → workload → pod.
+
+### Step 7 — Trace causality chains
+
+Use the evidence collected in phase 1 to trace the chain from symptom to root cause. If resource A fails because of B, investigate B. At each link, check status conditions and events. Common chains in OpenShift:
+- Pod pending → pod conditions (PodScheduled, Unschedulable) and events → node pressure, unschedulable nodes, resource quota, or PVC not bound
+- Pod crash-looping → container statuses (terminated reason: OOMKilled, Error) and logs → OOMKilled (check limits vs. actual usage), application error, misconfigured probes
+- Service unreachable → endpoints/endpointslices → no endpoints → pods not ready → readiness probe and pod conditions
+- Operator degraded → `bash tools/diagnose-operator.sh <name>` (check Available, Degraded, Progressing with messages) → operand pod failing → node issue or config error
+- Node NotReady → node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and events → kubelet issues, certificate expired, MCO update stuck, or kernel panic
+
+Use the range query data from step 2 and the recent changes from step 4 to identify **when** the problem started and what changed around that time.
+
+After identifying a root cause, keep digging: collect exact names, versions, and labels, and check for additional contributing factors before concluding.
+
+### Step 8 — Recommend remediation options
+
+Based on the collected evidence, propose remediation options. Provide the exact commands to run. When multiple options exist, list them from least to most disruptive and note whether each action is reversible. If the root cause is unclear, state what is known, suggest mitigations to reduce impact, and identify what additional data would narrow the diagnosis.
+
+**RBAC requirements for remediation:** the agent's ServiceAccount has **read-only** permissions (`cluster-reader` + `cluster-monitoring-view`). Any proposed remediation command (patch, delete, scale, rollout restart, etc.) requires additional RBAC that the agent does not have. For every proposed option, you **must** include a complete RBAC section listing:
+- The exact API group, resource, and verbs required (e.g., `apps` / `deployments` / `patch`)
+- The namespace scope (specific namespace or cluster-wide)
+- A ready-to-apply Role or ClusterRole YAML snippet the admin can use to grant the permissions
+
+This is critical; without this information the admin cannot safely execute the proposed remediation.

cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh

@@ -0,0 +1,86 @@
+#!/bin/bash
+# Detect recent changes that may have caused an issue: rollouts, image pulls,
+# and scaling events.
+#
+# Usage: check-recent-changes.sh [namespace] [minutes]
+#   namespace: target namespace (default: current). Cluster-level changes always included.
+#   minutes:   time window (default: 60)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+NAMESPACE="${1:-}"
+MINUTES="${2:-60}"
+
+NS_ARGS=()
+if [[ -n "$NAMESPACE" ]]; then
+  NS_ARGS=(-n "$NAMESPACE")
+fi
+
+# Deployment rollout status in namespace
+DEPLOYMENTS="[]"
+if [[ -n "$NAMESPACE" ]]; then
+  DEPLOY_JSON=$(oc get deployments "${NS_ARGS[@]}" -o json 2>/dev/null) || DEPLOY_JSON='{"items":[]}'
+  require_json "$DEPLOY_JSON" "deployment list"
+  DEPLOYMENTS=$(echo "$DEPLOY_JSON" | jq '[.items[] | select(
+      (.status.conditions[]? | select(.type=="Progressing" and .status=="True" and (.reason? == "NewReplicaSetAvailable" or .reason? == "ReplicaSetUpdated")))
+      or (.status.unavailableReplicas // 0) > 0
+    ) | {
+      name: .metadata.name,
+      namespace: .metadata.namespace,
+      ready: "\(.status.readyReplicas // 0)/\(.spec.replicas // 0)",
+      updated: (.status.updatedReplicas // 0),
+      unavailable: (.status.unavailableReplicas // 0),
+      last_update: (.status.conditions[] | select(.type=="Progressing") | .lastUpdateTime)
+    }]' 2>/dev/null || echo "[]")
+fi
+
+CUTOFF=$(compute_cutoff "$MINUTES")
+
+# Fetch events once, filter into rollout and image-pull categories
+ALL_EVENTS=$(oc get events "${NS_ARGS[@]}" --sort-by='.lastTimestamp' -o json 2>/dev/null) || \
+  ALL_EVENTS='{"items":[]}'
+require_json "$ALL_EVENTS" "events"
+
+ROLLOUT_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+    | select(.reason | test("ScalingReplicaSet|DeploymentRollback|Scheduled|Pulled|Created|Started|Killing"; "i"))
+    | {
+        time: .lastTimestamp,
+        reason: .reason,
+        object: "\(.involvedObject.kind)/\(.involvedObject.name)",
+        namespace: .involvedObject.namespace,
+        message: .message
+      }
+  ]' 2>/dev/null || echo "[]")
+
+IMAGE_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+    | select(.reason == "Pulled" or .reason == "Pulling")
+    | {
+        time: .lastTimestamp,
+        object: "\(.involvedObject.kind)/\(.involvedObject.name)",
+        namespace: .involvedObject.namespace,
+        message: .message
+      }
+  ]' 2>/dev/null || echo "[]")
+
+jq -n \
+  --arg ns "${NAMESPACE:-cluster}" \
+  --arg minutes "$MINUTES" \
+  --argjson rollouts "$ROLLOUT_EVENTS" \
+  --argjson images "$IMAGE_EVENTS" \
+  --argjson deployments "$DEPLOYMENTS" \
+  '{
+    namespace: $ns,
+    time_window_minutes: ($minutes | tonumber),
+    rollout_events: $rollouts,
+    image_pull_events: $images,
+    active_rollouts: $deployments
+  }'

cluster-troubleshoot/investigate-alert/tools/common.sh

@@ -0,0 +1,125 @@
+#!/bin/bash
+# Shared functions for investigate-alert tools.
+# Source this file: . "$(dirname "$0")/common.sh"
+
+set -euo pipefail
+
+# --- JSON output helpers ---
+
+error_json() {
+  local code="$1"
+  local message="$2"
+  local suggestion="${3:-}"
+  if [[ -n "$suggestion" ]]; then
+    jq -n --arg c "$code" --arg m "$message" --arg s "$suggestion" \
+      '{error:true, code:$c, message:$m, suggestion:$s}'
+  else
+    jq -n --arg c "$code" --arg m "$message" \
+      '{error:true, code:$c, message:$m}'
+  fi
+  exit 1
+}
+
+# --- Argument and environment validation ---
+
+require_command() {
+  local cmd="$1"
+  if ! command -v "$cmd" &>/dev/null; then
+    error_json "MISSING_TOOL" "$cmd is not available on PATH" "Install $cmd or verify the container image includes it"
+  fi
+}
+
+require_env() {
+  local var_name="$1"
+  if [[ -z "${!var_name:-}" ]]; then
+    error_json "NOT_CONFIGURED" "$var_name is not set" "Run: eval \$(bash tools/prometheus-setup.sh)"
+  fi
+}
+
+require_arg() {
+  local value="$1"
+  local name="$2"
+  local usage="${3:-}"
+  if [[ -z "$value" ]]; then
+    if [[ -n "$usage" ]]; then
+      error_json "MISSING_ARG" "$name is required" "$usage"
+    else
+      error_json "MISSING_ARG" "$name is required"
+    fi
+  fi
+}
+
+# --- Cluster access ---
+
+check_cluster_access() {
+  if ! oc whoami &>/dev/null; then
+    error_json "AUTH_FAILED" "Cannot authenticate to cluster" "Run: oc login <cluster-url>"
+  fi
+}
+
+# --- URL encoding ---
+
+urlencode() {
+  printf '%s' "$1" | python3 -c "import sys, urllib.parse; print(urllib.parse.quote(sys.stdin.read()))"
+}
+
+# --- Date helpers ---
+
+compute_cutoff() {
+  local minutes="$1"
+  date -u -d "${minutes} minutes ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || \
+    date -u -d "@$(($(date +%s) - minutes * 60))" +%Y-%m-%dT%H:%M:%SZ
+}
+
+# --- JSON validation ---
+
+require_json() {
+  local input="$1"
+  local context="${2:-API response}"
+  if ! echo "$input" | jq empty 2>/dev/null; then
+    error_json "INVALID_JSON" "Failed to parse ${context} as JSON" \
+      "The cluster API returned unexpected output"
+  fi
+}
+
+# --- Prometheus helpers ---
+
+prometheus_get() {
+  local path="$1"
+  require_env TOKEN
+  require_env THANOS_URL
+
+  local http_code
+  local tmpfile
+  local stderr_file
+  tmpfile=$(mktemp)
+  stderr_file=$(mktemp)
+  trap 'rm -f "$tmpfile" "$stderr_file"' RETURN
+
+  wget -qO "$tmpfile" --no-check-certificate \
+    --header="Authorization: Bearer $TOKEN" \
+    --server-response \
+    "https://${THANOS_URL}${path}" 2>"$stderr_file" || true
+
+  http_code=$(grep "HTTP/" "$stderr_file" 2>/dev/null | tail -1 | awk '{print $2}') || true
+
+  if [[ ! -s "$tmpfile" ]]; then
+    error_json "PROMETHEUS_ERROR" "No response from Prometheus at ${THANOS_URL}${path}" \
+      "Verify TOKEN and THANOS_URL with: eval \$(bash tools/prometheus-setup.sh)"
+  fi
+
+  if [[ -n "$http_code" && "$http_code" -ge 400 ]] 2>/dev/null; then
+    local body
+    body=$(cat "$tmpfile")
+    case "$http_code" in
+      401) error_json "AUTH_EXPIRED" "Prometheus returned 401 Unauthorized" \
+             "Token may have expired. Re-run: eval \$(bash tools/prometheus-setup.sh)" ;;
+      403) error_json "FORBIDDEN" "Prometheus returned 403 Forbidden" \
+             "ServiceAccount lacks cluster-monitoring-view ClusterRoleBinding" ;;
+      *)   error_json "PROMETHEUS_HTTP_${http_code}" "Prometheus returned HTTP ${http_code}" \
+             "Response: ${body:0:200}" ;;
+    esac
+  fi
+
+  cat "$tmpfile"
+}