diff --git a/cluster-troubleshoot/OWNERS b/cluster-troubleshoot/OWNERS
new file mode 100644
index 0000000..56f9e98
--- /dev/null
+++ b/cluster-troubleshoot/OWNERS
@@ -0,0 +1,16 @@
+# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md
+
+approvers:
+  - falox
+  - tremes
+  - rioloc
+  - shwetaap
+  - iNecas
+reviewers:
+  - falox
+  - tremes
+  - rioloc
+  - shwetaap
+  - iNecas
+
+component: "Cluster Troubleshooting Lightspeed Skills"
diff --git a/cluster-troubleshoot/README.md b/cluster-troubleshoot/README.md
new file mode 100644
index 0000000..c5d37c7
--- /dev/null
+++ b/cluster-troubleshoot/README.md
@@ -0,0 +1,3 @@
+# Cluster Troubleshooting skills
+
+This directory contains skills which are designed to help agents diagnose and troubleshoot OpenShift cluster issues.
diff --git a/cluster-troubleshoot/investigate-alert/SKILL.md b/cluster-troubleshoot/investigate-alert/SKILL.md
new file mode 100644
index 0000000..0e76104
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/SKILL.md
@@ -0,0 +1,142 @@
+---
+name: investigate-alert
+description: Investigate one or more firing OpenShift alerts to determine root cause and recommend remediation options. Use when the user provides alert names or a set of firing alerts. Not for cluster setup, configuration how-tos, or writing alerting rules.
+---
+
+# Environment
+
+- Platform: OpenShift Container Platform (OCP). Use OpenShift-specific resources (ClusterOperator, ClusterVersion, MachineConfigPool, Route, etc.) alongside standard Kubernetes ones.
+- You run commands against the live cluster.
+- Available CLIs: oc, kubectl, jq, wget, openssl, skopeo, python3, dig, nslookup, ip, ss, tcpdump, strace
+- Writable directories: /home/agent, /tmp/agent-workspace. The root filesystem is read-only.
+
+# Rules
+
+- Verify every claim with evidence from command output. Provide exact resource names, namespaces, timestamps, and error messages.
+- If multiple causes exist, list them numbered with supporting evidence.
+- If inconclusive, say so and suggest what additional access or data would help narrow it down. Never fabricate information.
+- Stay focused on the reported alerts; don't surface unrelated errors.
+- The remediation options must address only the root cause of the alerts being analyzed. Never include secondary issues, unrelated findings, or general recommendations.
+- No URLs unless from command output or provided context.
+- List actual resources before inspecting them; never guess pod names, use label selectors or owner references.
+- Sample up to 3 representative pods per workload, not all.
+- All `tools/` scripts output JSON except `prometheus-setup.sh` (shell export commands for `eval`). Use `jq` for further filtering when needed.
+- When running raw `oc` commands, use `-o json | jq` for structured data extraction.
+- Do not repeat the same command with the same arguments.
+- Do not ask the user to run a command; gather the information yourself.
+- Be highly concise. Evidence-backed conclusions, no filler.
+
+# Tools
+
+The `tools/` directory contains diagnostic scripts. All output JSON to stdout and return structured error objects on failure, except `prometheus-setup.sh` which prints shell export commands for `eval`. Run `eval $(bash tools/prometheus-setup.sh)` once per session before using Prometheus tools.
+
+## Prometheus
+
+- `bash tools/prometheus-setup.sh` — Set TOKEN and THANOS_URL
+- `bash tools/query-alert.sh <alert_name>` — Firing instances of a named alert with full label sets
+- `bash tools/fetch-alert-rule.sh <alert_name>` — Alert PromQL expression, thresholds, annotations
+- `bash tools/prometheus-query.sh '<promql>'` — Instant PromQL query
+- `bash tools/prometheus-query-range.sh '<promql>' [duration] [step]` — Range query (default: 1h, 60s step)
+- `bash tools/discover-metrics.sh <pattern>` — Search metric names by pattern
+- `bash tools/get-firing-alerts.sh [filter]` — All firing alerts, optionally filtered by name
+
+## Cluster
+
+- `bash tools/diagnose-pod.sh <name-or-selector> [namespace] [--logs]` — Pod status, conditions, events, logs. Selectors (containing `=`) sample up to 3 pods.
+- `bash tools/diagnose-node.sh [node_name]` — Node conditions, capacity, taints. Without a name: all nodes summary.
+- `bash tools/diagnose-operator.sh [operator_name]` — ClusterOperator health. Without a name: all operators summary.
+- `bash tools/diagnose-workload.sh <name> [namespace] [kind]` — Deployment/StatefulSet/DaemonSet status, replicas, rollout history. Auto-detects kind.
+- `bash tools/get-events.sh [namespace] [minutes]` — Events sorted by time, last N minutes (default: 60). Use `--all` for cluster-wide.
+- `bash tools/check-recent-changes.sh [namespace] [minutes]` — Recent rollouts, image pulls, active rollouts.
+
+# Investigation Protocol
+
+The protocol has two phases: **collect evidence first**, then **analyze**. Do not jump to conclusions or propose remediation until you have completed the collection phase.
+
+## Phase 1 — Collect evidence
+
+### Step 1 — Scope the blast radius
+
+Determine the scope of the alerts: is it one pod, one node, one namespace, or cluster-wide? This determines which layer to start from and which tools to prioritize.
+
+### Step 2 — Set up and query the alerts
+
+Configure Prometheus access:
+```bash
+eval $(bash tools/prometheus-setup.sh)
+```
+
+For each alert provided:
+```bash
+bash tools/query-alert.sh <ALERT_NAME>
+bash tools/fetch-alert-rule.sh <ALERT_NAME>
+```
+Extract the full label set (namespace, pod, node, service, severity, etc.) from firing instances. Even if an alert is NOT firing, fetch the rule; the PromQL expression and labels contain diagnostic information.
+
+If the rule is found, check the annotations for `runbook_url`. If present, extract the URL and follow the runbook steps as part of the analysis.
+
+Run each alert's PromQL expression to see current and recent values:
+```bash
+bash tools/prometheus-query.sh '<expr>'
+bash tools/prometheus-query-range.sh '<expr>'
+```
+
+### Step 3 — Collect workload state and logs
+
+Using the namespaces and resources identified from the alert labels, list all deployments, statefulsets, and daemonsets in the affected namespaces. Then for **every** workload, collect its status and logs:
+```bash
+bash tools/diagnose-workload.sh <name> <namespace>
+bash tools/diagnose-pod.sh <name-or-selector> <namespace> --logs
+```
+This is mandatory. Do not skip workloads that appear healthy. A Running pod with no visible issues can still be the root cause if it is producing errors internally.
+
+### Step 4 — Collect events and recent changes
+
+```bash
+bash tools/get-events.sh <namespace>
+bash tools/check-recent-changes.sh <namespace>
+```
+
+The scripts cover rollouts and image pulls. Also check for: config/secret edits, HPA scaling events, operator upgrades, node drains, and MachineConfig updates using `oc rollout history`, `oc describe`, and events sorted by time.
+
+### Step 5 — Collect all firing alerts for correlation
+
+```bash
+bash tools/get-firing-alerts.sh
+```
+
+## Phase 2 — Analyze and diagnose
+
+### Step 6 — Correlate alerts
+
+Group all firing alerts by shared labels to find the common root cause:
+- Alerts sharing the same **node** → likely a node-level issue (investigate the node first)
+- Alerts sharing the same **namespace** but on different nodes → likely an application or config issue
+- Alerts sharing the same **job** or **service** → dependency or networking issue
+- Alerts across multiple namespaces and nodes → cluster-wide issue (check operators, control plane, networking)
+
+Prioritize investigation from infrastructure inward: node → operator → workload → pod.
+
+### Step 7 — Trace causality chains
+
+Use the evidence collected in phase 1 to trace the chain from symptom to root cause. If resource A fails because of B, investigate B. At each link, check status conditions and events. Common chains in OpenShift:
+- Pod pending → pod conditions (PodScheduled, Unschedulable) and events → node pressure, unschedulable nodes, resource quota, or PVC not bound
+- Pod crash-looping → container statuses (terminated reason: OOMKilled, Error) and logs → OOMKilled (check limits vs. actual usage), application error, misconfigured probes
+- Service unreachable → endpoints/endpointslices → no endpoints → pods not ready → readiness probe and pod conditions
+- Operator degraded → `bash tools/diagnose-operator.sh <name>` (check Available, Degraded, Progressing with messages) → operand pod failing → node issue or config error
+- Node NotReady → node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and events → kubelet issues, certificate expired, MCO update stuck, or kernel panic
+
+Use the range query data from step 2 and the recent changes from step 4 to identify **when** the problem started and what changed around that time.
+
+After identifying a root cause, keep digging: collect exact names, versions, and labels, and check for additional contributing factors before concluding.
+
+### Step 8 — Recommend remediation options
+
+Based on the collected evidence, propose remediation options. Provide the exact commands to run. When multiple options exist, list them from least to most disruptive and note whether each action is reversible. If the root cause is unclear, state what is known, suggest mitigations to reduce impact, and identify what additional data would narrow the diagnosis.
+
+**RBAC requirements for remediation:** the agent's ServiceAccount has **read-only** permissions (`cluster-reader` + `cluster-monitoring-view`). Any proposed remediation command (patch, delete, scale, rollout restart, etc.) requires additional RBAC that the agent does not have. For every proposed option, you **must** include a complete RBAC section listing:
+- The exact API group, resource, and verbs required (e.g., `apps` / `deployments` / `patch`)
+- The namespace scope (specific namespace or cluster-wide)
+- A ready-to-apply Role or ClusterRole YAML snippet the admin can use to grant the permissions
+
+This is critical; without this information the admin cannot safely execute the proposed remediation.
diff --git a/cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh b/cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh
new file mode 100755
index 0000000..245a5bc
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/check-recent-changes.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Detect recent changes that may have caused an issue: rollouts, image pulls,
+# and scaling events.
+#
+# Usage: check-recent-changes.sh [namespace] [minutes]
+#   namespace: target namespace (default: current). Cluster-level changes always included.
+#   minutes:   time window (default: 60)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+NAMESPACE="${1:-}"
+MINUTES="${2:-60}"
+
+NS_ARGS=()
+if [[ -n "$NAMESPACE" ]]; then
+  NS_ARGS=(-n "$NAMESPACE")
+fi
+
+# Deployment rollout status in namespace
+DEPLOYMENTS="[]"
+if [[ -n "$NAMESPACE" ]]; then
+  DEPLOY_JSON=$(oc get deployments "${NS_ARGS[@]}" -o json 2>/dev/null) || DEPLOY_JSON='{"items":[]}'
+  require_json "$DEPLOY_JSON" "deployment list"
+  DEPLOYMENTS=$(echo "$DEPLOY_JSON" | jq '[.items[] | select(
+      (.status.conditions[]? | select(.type=="Progressing" and .status=="True" and (.reason? == "NewReplicaSetAvailable" or .reason? == "ReplicaSetUpdated")))
+      or (.status.unavailableReplicas // 0) > 0
+    ) | {
+      name: .metadata.name,
+      namespace: .metadata.namespace,
+      ready: "\(.status.readyReplicas // 0)/\(.spec.replicas // 0)",
+      updated: (.status.updatedReplicas // 0),
+      unavailable: (.status.unavailableReplicas // 0),
+      last_update: (.status.conditions[] | select(.type=="Progressing") | .lastUpdateTime)
+    }]' 2>/dev/null || echo "[]")
+fi
+
+CUTOFF=$(compute_cutoff "$MINUTES")
+
+# Fetch events once, filter into rollout and image-pull categories
+ALL_EVENTS=$(oc get events "${NS_ARGS[@]}" --sort-by='.lastTimestamp' -o json 2>/dev/null) || \
+  ALL_EVENTS='{"items":[]}'
+require_json "$ALL_EVENTS" "events"
+
+ROLLOUT_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+    | select(.reason | test("ScalingReplicaSet|DeploymentRollback|Scheduled|Pulled|Created|Started|Killing"; "i"))
+    | {
+        time: .lastTimestamp,
+        reason: .reason,
+        object: "\(.involvedObject.kind)/\(.involvedObject.name)",
+        namespace: .involvedObject.namespace,
+        message: .message
+      }
+  ]' 2>/dev/null || echo "[]")
+
+IMAGE_EVENTS=$(echo "$ALL_EVENTS" | jq --arg cutoff "$CUTOFF" '[
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+    | select(.reason == "Pulled" or .reason == "Pulling")
+    | {
+        time: .lastTimestamp,
+        object: "\(.involvedObject.kind)/\(.involvedObject.name)",
+        namespace: .involvedObject.namespace,
+        message: .message
+      }
+  ]' 2>/dev/null || echo "[]")
+
+jq -n \
+  --arg ns "${NAMESPACE:-cluster}" \
+  --arg minutes "$MINUTES" \
+  --argjson rollouts "$ROLLOUT_EVENTS" \
+  --argjson images "$IMAGE_EVENTS" \
+  --argjson deployments "$DEPLOYMENTS" \
+  '{
+    namespace: $ns,
+    time_window_minutes: ($minutes | tonumber),
+    rollout_events: $rollouts,
+    image_pull_events: $images,
+    active_rollouts: $deployments
+  }'
diff --git a/cluster-troubleshoot/investigate-alert/tools/common.sh b/cluster-troubleshoot/investigate-alert/tools/common.sh
new file mode 100755
index 0000000..91c5a4b
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/common.sh
@@ -0,0 +1,125 @@
+#!/bin/bash
+# Shared functions for investigate-alert tools.
+# Source this file: . "$(dirname "$0")/common.sh"
+
+set -euo pipefail
+
+# --- JSON output helpers ---
+
+error_json() {
+  local code="$1"
+  local message="$2"
+  local suggestion="${3:-}"
+  if [[ -n "$suggestion" ]]; then
+    jq -n --arg c "$code" --arg m "$message" --arg s "$suggestion" \
+      '{error:true, code:$c, message:$m, suggestion:$s}'
+  else
+    jq -n --arg c "$code" --arg m "$message" \
+      '{error:true, code:$c, message:$m}'
+  fi
+  exit 1
+}
+
+# --- Argument and environment validation ---
+
+require_command() {
+  local cmd="$1"
+  if ! command -v "$cmd" &>/dev/null; then
+    error_json "MISSING_TOOL" "$cmd is not available on PATH" "Install $cmd or verify the container image includes it"
+  fi
+}
+
+require_env() {
+  local var_name="$1"
+  if [[ -z "${!var_name:-}" ]]; then
+    error_json "NOT_CONFIGURED" "$var_name is not set" "Run: eval \$(bash tools/prometheus-setup.sh)"
+  fi
+}
+
+require_arg() {
+  local value="$1"
+  local name="$2"
+  local usage="${3:-}"
+  if [[ -z "$value" ]]; then
+    if [[ -n "$usage" ]]; then
+      error_json "MISSING_ARG" "$name is required" "$usage"
+    else
+      error_json "MISSING_ARG" "$name is required"
+    fi
+  fi
+}
+
+# --- Cluster access ---
+
+check_cluster_access() {
+  if ! oc whoami &>/dev/null; then
+    error_json "AUTH_FAILED" "Cannot authenticate to cluster" "Run: oc login <cluster-url>"
+  fi
+}
+
+# --- URL encoding ---
+
+urlencode() {
+  printf '%s' "$1" | python3 -c "import sys, urllib.parse; print(urllib.parse.quote(sys.stdin.read()))"
+}
+
+# --- Date helpers ---
+
+compute_cutoff() {
+  local minutes="$1"
+  date -u -d "${minutes} minutes ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || \
+    date -u -d "@$(($(date +%s) - minutes * 60))" +%Y-%m-%dT%H:%M:%SZ
+}
+
+# --- JSON validation ---
+
+require_json() {
+  local input="$1"
+  local context="${2:-API response}"
+  if ! echo "$input" | jq empty 2>/dev/null; then
+    error_json "INVALID_JSON" "Failed to parse ${context} as JSON" \
+      "The cluster API returned unexpected output"
+  fi
+}
+
+# --- Prometheus helpers ---
+
+prometheus_get() {
+  local path="$1"
+  require_env TOKEN
+  require_env THANOS_URL
+
+  local http_code
+  local tmpfile
+  local stderr_file
+  tmpfile=$(mktemp)
+  stderr_file=$(mktemp)
+  trap 'rm -f "$tmpfile" "$stderr_file"' RETURN
+
+  wget -qO "$tmpfile" --no-check-certificate \
+    --header="Authorization: Bearer $TOKEN" \
+    --server-response \
+    "https://${THANOS_URL}${path}" 2>"$stderr_file" || true
+
+  http_code=$(grep "HTTP/" "$stderr_file" 2>/dev/null | tail -1 | awk '{print $2}') || true
+
+  if [[ ! -s "$tmpfile" ]]; then
+    error_json "PROMETHEUS_ERROR" "No response from Prometheus at ${THANOS_URL}${path}" \
+      "Verify TOKEN and THANOS_URL with: eval \$(bash tools/prometheus-setup.sh)"
+  fi
+
+  if [[ -n "$http_code" && "$http_code" -ge 400 ]] 2>/dev/null; then
+    local body
+    body=$(cat "$tmpfile")
+    case "$http_code" in
+      401) error_json "AUTH_EXPIRED" "Prometheus returned 401 Unauthorized" \
+             "Token may have expired. Re-run: eval \$(bash tools/prometheus-setup.sh)" ;;
+      403) error_json "FORBIDDEN" "Prometheus returned 403 Forbidden" \
+             "ServiceAccount lacks cluster-monitoring-view ClusterRoleBinding" ;;
+      *)   error_json "PROMETHEUS_HTTP_${http_code}" "Prometheus returned HTTP ${http_code}" \
+             "Response: ${body:0:200}" ;;
+    esac
+  fi
+
+  cat "$tmpfile"
+}
diff --git a/cluster-troubleshoot/investigate-alert/tools/diagnose-node.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-node.sh
new file mode 100755
index 0000000..6bc4ca3
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/diagnose-node.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# Node diagnostics: conditions, capacity, allocatable, taints, kubelet info.
+#
+# Usage: diagnose-node.sh [node_name]
+#   If no node name given, shows all nodes with non-Ready conditions.
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+NODE_NAME="${1:-}"
+
+if [[ -n "$NODE_NAME" ]]; then
+  NODE_JSON=$(oc get node "$NODE_NAME" -o json 2>/dev/null) || \
+    error_json "NOT_FOUND" "Node '$NODE_NAME' not found" "List nodes with: oc get nodes"
+  require_json "$NODE_JSON" "node $NODE_NAME"
+
+  echo "$NODE_JSON" | jq '{
+    nodes: [{
+      name: .metadata.name,
+      labels: (.metadata.labels | with_entries(select(.key | test("node-role|kubernetes.io/os|topology")))),
+      conditions: .status.conditions,
+      capacity: .status.capacity,
+      allocatable: .status.allocatable,
+      taints: (.spec.taints // []),
+      unschedulable: (.spec.unschedulable // false),
+      kubelet_version: .status.nodeInfo.kubeletVersion,
+      os_image: .status.nodeInfo.osImage,
+      container_runtime: .status.nodeInfo.containerRuntimeVersion
+    }]
+  }'
+else
+  # Show all nodes, flagging unhealthy ones
+  NODES_JSON=$(oc get nodes -o json 2>/dev/null) || \
+    error_json "CLUSTER_ERROR" "Cannot list nodes"
+  require_json "$NODES_JSON" "node list"
+
+  echo "$NODES_JSON" | jq '{
+    nodes: [.items[] | {
+      name: .metadata.name,
+      labels: (.metadata.labels | with_entries(select(.key | test("node-role|kubernetes.io/os|topology")))),
+      ready: (.status.conditions[] | select(.type=="Ready") | .status == "True"),
+      conditions: [.status.conditions[] | select(.status != "False" or .type == "Ready")],
+      taints: (.spec.taints // []),
+      unschedulable: (.spec.unschedulable // false),
+      kubelet_version: .status.nodeInfo.kubeletVersion
+    }],
+    summary: {
+      total: (.items | length),
+      ready: [.items[] | select(.status.conditions[] | select(.type=="Ready" and .status=="True"))] | length,
+      not_ready: [.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True"))] | length
+    }
+  }'
+fi
diff --git a/cluster-troubleshoot/investigate-alert/tools/diagnose-operator.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-operator.sh
new file mode 100755
index 0000000..b899abf
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/diagnose-operator.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+# ClusterOperator diagnostics: Available, Progressing, Degraded conditions.
+#
+# Usage: diagnose-operator.sh [operator_name]
+#   If no operator name given, lists all operators with non-healthy conditions.
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+OPERATOR_NAME="${1:-}"
+
+if [[ -n "$OPERATOR_NAME" ]]; then
+  CO_JSON=$(oc get clusteroperator "$OPERATOR_NAME" -o json 2>/dev/null) || \
+    error_json "NOT_FOUND" "ClusterOperator '$OPERATOR_NAME' not found" "List operators with: oc get clusteroperators"
+  require_json "$CO_JSON" "ClusterOperator $OPERATOR_NAME"
+
+  echo "$CO_JSON" | jq '{
+    operators: [{
+      name: .metadata.name,
+      available: (.status.conditions[] | select(.type=="Available") | .status == "True"),
+      progressing: (.status.conditions[] | select(.type=="Progressing") | .status == "True"),
+      degraded: (.status.conditions[] | select(.type=="Degraded") | .status == "True"),
+      versions: [.status.versions[]? | {(.name): .version}] | add // {},
+      conditions: .status.conditions
+    }]
+  }'
+else
+  CO_JSON=$(oc get clusteroperators -o json 2>/dev/null) || \
+    error_json "CLUSTER_ERROR" "Cannot list ClusterOperators"
+  require_json "$CO_JSON" "ClusterOperator list"
+
+  echo "$CO_JSON" | jq '{
+    operators: [.items[] | {
+      name: .metadata.name,
+      available: (.status.conditions[] | select(.type=="Available") | .status == "True"),
+      progressing: (.status.conditions[] | select(.type=="Progressing") | .status == "True"),
+      degraded: (.status.conditions[] | select(.type=="Degraded") | .status == "True"),
+      versions: [.status.versions[]? | {(.name): .version}] | add // {}
+    }],
+    summary: {
+      total: (.items | length),
+      available: [.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="True"))] | length,
+      degraded: [.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True"))] | length,
+      progressing: [.items[] | select(.status.conditions[] | select(.type=="Progressing" and .status=="True"))] | length
+    }
+  }'
+fi
diff --git a/cluster-troubleshoot/investigate-alert/tools/diagnose-pod.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-pod.sh
new file mode 100755
index 0000000..f576775
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/diagnose-pod.sh
@@ -0,0 +1,102 @@
+#!/bin/bash
+# Pod diagnostics: status, conditions, container statuses, events, and logs.
+#
+# Usage: diagnose-pod.sh <name-or-selector> [namespace] [--logs]
+#   If <name-or-selector> contains '=', it is treated as a label selector.
+#   Samples up to 3 pods when using a label selector.
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+require_arg "${1:-}" "pod name or label selector" "Usage: diagnose-pod.sh <name-or-selector> [namespace] [--logs]"
+
+TARGET="$1"
+NAMESPACE="${2:-}"
+INCLUDE_LOGS=false
+
+# Parse flags from remaining args
+for arg in "$@"; do
+  if [[ "$arg" == "--logs" ]]; then
+    INCLUDE_LOGS=true
+  fi
+done
+
+NS_ARGS=()
+if [[ -n "$NAMESPACE" && "$NAMESPACE" != "--logs" ]]; then
+  NS_ARGS=(-n "$NAMESPACE")
+fi
+
+# Determine if target is a label selector or pod name
+if [[ "$TARGET" == *"="* ]]; then
+  POD_JSON=$(oc get pods -l "$TARGET" "${NS_ARGS[@]}" -o json 2>/dev/null) || \
+    error_json "NOT_FOUND" "No pods matching selector '$TARGET'" "Verify the label selector and namespace"
+  require_json "$POD_JSON" "pod list"
+  POD_NAMES=$(echo "$POD_JSON" | jq -r '.items[].metadata.name' | head -3)
+else
+  POD_JSON=$(oc get pod "$TARGET" "${NS_ARGS[@]}" -o json 2>/dev/null) || \
+    error_json "NOT_FOUND" "Pod '$TARGET' not found" "Check the pod name and namespace"
+  require_json "$POD_JSON" "pod"
+  POD_NAMES="$TARGET"
+fi
+
+if [[ -z "$POD_NAMES" ]]; then
+  error_json "NOT_FOUND" "No pods found matching '$TARGET'"
+fi
+
+RESULTS="[]"
+
+while IFS= read -r POD_NAME; do
+  [[ -z "$POD_NAME" ]] && continue
+
+  # Get pod details
+  POD_DETAIL=$(oc get pod "$POD_NAME" "${NS_ARGS[@]}" -o json 2>/dev/null) || continue
+  require_json "$POD_DETAIL" "pod $POD_NAME"
+  POD_NS=$(echo "$POD_DETAIL" | jq -r '.metadata.namespace')
+
+  # Extract key fields
+  POD_INFO=$(echo "$POD_DETAIL" | jq '{
+    name: .metadata.name,
+    namespace: .metadata.namespace,
+    phase: .status.phase,
+    conditions: .status.conditions,
+    container_statuses: .status.containerStatuses,
+    init_container_statuses: .status.initContainerStatuses,
+    owner_references: .metadata.ownerReferences,
+    node_name: .spec.nodeName,
+    restart_count: ([.status.containerStatuses[]?.restartCount] | add // 0)
+  }')
+
+  # Get events for this pod
+  EVENTS=$(oc get events -n "$POD_NS" \
+    --field-selector "involvedObject.name=$POD_NAME" \
+    --sort-by='.lastTimestamp' -o json 2>/dev/null | \
+    jq '[.items[-10:] | .[] | {
+      time: .lastTimestamp,
+      type: .type,
+      reason: .reason,
+      message: .message,
+      count: .count
+    }]' 2>/dev/null || echo "[]")
+
+  POD_INFO=$(echo "$POD_INFO" | jq --argjson events "$EVENTS" '. + {events: $events}')
+
+  # Get logs if requested
+  if [[ "$INCLUDE_LOGS" == "true" ]]; then
+    CONTAINERS=$(echo "$POD_DETAIL" | jq -r '.spec.containers[].name')
+    LOGS_OBJ="{}"
+    while IFS= read -r CONTAINER; do
+      [[ -z "$CONTAINER" ]] && continue
+      CONTAINER_LOGS=$(oc logs "$POD_NAME" -n "$POD_NS" -c "$CONTAINER" --tail=50 2>/dev/null || echo "(logs unavailable)")
+      LOGS_ARRAY=$(echo "$CONTAINER_LOGS" | jq -R . | jq -s .)
+      LOGS_OBJ=$(echo "$LOGS_OBJ" | jq --arg c "$CONTAINER" --argjson l "$LOGS_ARRAY" '. + {($c): $l}')
+    done <<< "$CONTAINERS"
+    POD_INFO=$(echo "$POD_INFO" | jq --argjson logs "$LOGS_OBJ" '. + {logs: $logs}')
+  fi
+
+  RESULTS=$(echo "$RESULTS" | jq --argjson pod "$POD_INFO" '. + [$pod]')
+done <<< "$POD_NAMES"
+
+echo "$RESULTS" | jq '{pods: .}'
diff --git a/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh b/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh
new file mode 100755
index 0000000..2f5424d
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/diagnose-workload.sh
@@ -0,0 +1,106 @@
+#!/bin/bash
+# Workload diagnostics: Deployment, StatefulSet, or DaemonSet status,
+# replica counts, conditions, and rollout history.
+#
+# Usage: diagnose-workload.sh <name> [namespace] [kind]
+#   kind: deployment, statefulset, daemonset (default: auto-detect)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+require_arg "${1:-}" "workload name" "Usage: diagnose-workload.sh <name> [namespace] [kind]"
+
+WORKLOAD_NAME="$1"
+NAMESPACE="${2:-}"
+KIND="${3:-}"
+
+NS_ARGS=()
+if [[ -n "$NAMESPACE" ]]; then
+  NS_ARGS=(-n "$NAMESPACE")
+fi
+
+# Auto-detect kind if not specified
+if [[ -z "$KIND" ]]; then
+  for try_kind in deployment statefulset daemonset; do
+    if oc get "$try_kind" "$WORKLOAD_NAME" "${NS_ARGS[@]}" &>/dev/null; then
+      KIND="$try_kind"
+      break
+    fi
+  done
+  if [[ -z "$KIND" ]]; then
+    error_json "NOT_FOUND" "No deployment, statefulset, or daemonset named '$WORKLOAD_NAME' found" \
+      "Verify the name and namespace"
+  fi
+fi
+
+WL_JSON=$(oc get "$KIND" "$WORKLOAD_NAME" "${NS_ARGS[@]}" -o json 2>/dev/null) || \
+  error_json "NOT_FOUND" "$KIND '$WORKLOAD_NAME' not found"
+require_json "$WL_JSON" "$KIND $WORKLOAD_NAME"
+
+# Get rollout history
+HISTORY=$(oc rollout history "$KIND/$WORKLOAD_NAME" "${NS_ARGS[@]}" 2>/dev/null || echo "")
+
+# Build output based on kind
+case "$KIND" in
+  deployment)
+    echo "$WL_JSON" | jq --arg hist "$HISTORY" '{
+      kind: "Deployment",
+      name: .metadata.name,
+      namespace: .metadata.namespace,
+      replicas: {
+        desired: (.spec.replicas // 0),
+        ready: (.status.readyReplicas // 0),
+        available: (.status.availableReplicas // 0),
+        updated: (.status.updatedReplicas // 0),
+        unavailable: (.status.unavailableReplicas // 0)
+      },
+      strategy: .spec.strategy.type,
+      conditions: .status.conditions,
+      containers: [.spec.template.spec.containers[] | {name, image}],
+      rollout_history: $hist
+    }'
+    ;;
+  statefulset)
+    echo "$WL_JSON" | jq --arg hist "$HISTORY" '{
+      kind: "StatefulSet",
+      name: .metadata.name,
+      namespace: .metadata.namespace,
+      replicas: {
+        desired: (.spec.replicas // 0),
+        ready: (.status.readyReplicas // 0),
+        current: (.status.currentReplicas // 0),
+        updated: (.status.updatedReplicas // 0)
+      },
+      update_strategy: .spec.updateStrategy.type,
+      conditions: .status.conditions,
+      containers: [.spec.template.spec.containers[] | {name, image}],
+      volume_claims: [.spec.volumeClaimTemplates[]? | .metadata.name],
+      rollout_history: $hist
+    }'
+    ;;
+  daemonset)
+    echo "$WL_JSON" | jq --arg hist "$HISTORY" '{
+      kind: "DaemonSet",
+      name: .metadata.name,
+      namespace: .metadata.namespace,
+      replicas: {
+        desired: (.status.desiredNumberScheduled // 0),
+        ready: (.status.numberReady // 0),
+        available: (.status.numberAvailable // 0),
+        misscheduled: (.status.numberMisscheduled // 0),
+        unavailable: (.status.numberUnavailable // 0)
+      },
+      update_strategy: .spec.updateStrategy.type,
+      conditions: .status.conditions,
+      containers: [.spec.template.spec.containers[] | {name, image}],
+      rollout_history: $hist
+    }'
+    ;;
+  *)
+    error_json "UNSUPPORTED_KIND" "Unsupported workload kind: $KIND" \
+      "Supported kinds: deployment, statefulset, daemonset"
+    ;;
+esac
diff --git a/cluster-troubleshoot/investigate-alert/tools/discover-metrics.sh b/cluster-troubleshoot/investigate-alert/tools/discover-metrics.sh
new file mode 100755
index 0000000..c901df1
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/discover-metrics.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# Search for Prometheus metric names matching a pattern.
+#
+# Usage: discover-metrics.sh <pattern>
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+require_arg "${1:-}" "search pattern" "Usage: discover-metrics.sh <pattern>"
+
+PATTERN="$1"
+
+ALL_METRICS=$(prometheus_get "/api/v1/label/__name__/values")
+MATCHED=$(echo "$ALL_METRICS" | jq -r '.data[]' | grep -i "$PATTERN" || true)
+
+if [[ -z "$MATCHED" ]]; then
+  jq -n --arg p "$PATTERN" '{pattern: $p, match_count: 0, metrics: []}'
+else
+  echo "$MATCHED" | jq -R . | jq -s --arg p "$PATTERN" \
+    '{pattern: $p, match_count: length, metrics: .}'
+fi
diff --git a/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh b/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh
new file mode 100755
index 0000000..8e51b7a
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/fetch-alert-rule.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# Fetch the alerting rule definition: PromQL expression, thresholds,
+# duration, labels, and annotations (description, summary, runbook_url).
+#
+# Usage: fetch-alert-rule.sh <alert_name>
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+require_arg "${1:-}" "alert name" "Usage: fetch-alert-rule.sh <alert_name>"
+
+ALERT_NAME="$1"
+
+RESPONSE=$(prometheus_get "/api/v1/rules?type=alert")
+RULE=$(echo "$RESPONSE" | jq --arg name "$ALERT_NAME" \
+  '[.data.groups[].rules[] | select(.name==$name and .type=="alerting")] | first // empty')
+
+if [[ -z "$RULE" || "$RULE" == "null" ]]; then
+  jq -n --arg name "$ALERT_NAME" '{alert_name: $name, found: false}'
+else
+  echo "$RULE" | jq --arg name "$ALERT_NAME" '{
+    alert_name: $name,
+    found: true,
+    expr: .query,
+    duration: .duration,
+    labels: .labels,
+    annotations: .annotations
+  }'
+fi
diff --git a/cluster-troubleshoot/investigate-alert/tools/get-events.sh b/cluster-troubleshoot/investigate-alert/tools/get-events.sh
new file mode 100755
index 0000000..d95bf08
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/get-events.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+# Get events sorted by time for a namespace or cluster-wide.
+#
+# Usage: get-events.sh [namespace] [minutes]
+#   namespace: target namespace, or '--all' for cluster-wide (default: current)
+#   minutes:   only show events from last N minutes (default: 60)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+NAMESPACE="${1:-}"
+MINUTES="${2:-60}"
+
+NS_ARGS=()
+if [[ "$NAMESPACE" == "--all" || "$NAMESPACE" == "-A" ]]; then
+  NS_ARGS=(-A)
+  NAMESPACE="all"
+elif [[ -n "$NAMESPACE" ]]; then
+  NS_ARGS=(-n "$NAMESPACE")
+fi
+
+CUTOFF=$(compute_cutoff "$MINUTES")
+
+EVENTS_JSON=$(oc get events "${NS_ARGS[@]}" --sort-by='.lastTimestamp' -o json 2>/dev/null) || \
+  error_json "CLUSTER_ERROR" "Cannot list events"
+require_json "$EVENTS_JSON" "events"
+
+# Filter to recent events and extract key fields
+echo "$EVENTS_JSON" | jq --arg cutoff "$CUTOFF" --arg ns "${NAMESPACE:-current}" --argjson minutes "$MINUTES" '{
+  namespace: $ns,
+  time_window_minutes: $minutes,
+  events: [
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+    | {
+        time: .lastTimestamp,
+        type: .type,
+        reason: .reason,
+        object: "\(.involvedObject.kind)/\(.involvedObject.name)",
+        namespace: .involvedObject.namespace,
+        message: .message,
+        count: .count
+      }
+  ] | .[-100:],
+  total_in_window: ([
+    .items[]
+    | select(.lastTimestamp >= $cutoff or .lastTimestamp == null)
+  ] | length)
+}'
diff --git a/cluster-troubleshoot/investigate-alert/tools/get-firing-alerts.sh b/cluster-troubleshoot/investigate-alert/tools/get-firing-alerts.sh
new file mode 100755
index 0000000..5cc761d
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/get-firing-alerts.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+# List all currently firing alerts from Prometheus.
+#
+# Usage: get-firing-alerts.sh [filter]
+#   filter: optional grep pattern to filter alert names
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+
+FILTER="${1:-}"
+
+RESPONSE=$(prometheus_get "/api/v1/alerts")
+
+if [[ -n "$FILTER" ]]; then
+  ALERTS=$(echo "$RESPONSE" | jq --arg f "$FILTER" \
+    '[.data.alerts[] | select(.state=="firing") | select(.labels.alertname | test($f; "i"))]')
+else
+  ALERTS=$(echo "$RESPONSE" | jq '[.data.alerts[] | select(.state=="firing")]')
+fi
+
+COUNT=$(echo "$ALERTS" | jq 'length')
+printf '{"total_firing":%d,"alerts":%s}\n' "$COUNT" "$ALERTS"
diff --git a/cluster-troubleshoot/investigate-alert/tools/prometheus-query-range.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-query-range.sh
new file mode 100755
index 0000000..0f92946
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/prometheus-query-range.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+# Run a PromQL range query with configurable time window and step.
+#
+# Usage: prometheus-query-range.sh '<promql>' [duration] [step]
+#   duration: lookback period (default: 1h). Supports: Ns, Nm, Nh, Nd
+#   step:     resolution (default: 60s)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+require_arg "${1:-}" "PromQL expression" "Usage: prometheus-query-range.sh '<promql>' [duration] [step]"
+
+PROMQL="$1"
+DURATION="${2:-1h}"
+STEP="${3:-60s}"
+
+# Parse duration to seconds
+parse_duration() {
+  local val="$1"
+  local num="${val%[smhd]}"
+  local unit="${val##*[0-9]}"
+  case "$unit" in
+    s) echo "$num" ;;
+    m) echo $((num * 60)) ;;
+    h) echo $((num * 3600)) ;;
+    d) echo $((num * 86400)) ;;
+    *) error_json "INVALID_ARG" "Invalid duration format: $val" "Use: Ns, Nm, Nh, or Nd (e.g. 30m, 1h, 1d)" ;;
+  esac
+}
+
+SECONDS_AGO=$(parse_duration "$DURATION")
+END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+START=$(date -u -d "${SECONDS_AGO} seconds ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null) || \
+  START=$(date -u -d "@$(($(date +%s) - SECONDS_AGO))" +%Y-%m-%dT%H:%M:%SZ)
+
+QUERY=$(urlencode "$PROMQL")
+prometheus_get "/api/v1/query_range?query=${QUERY}&start=${START}&end=${END}&step=${STEP}" | jq .
diff --git a/cluster-troubleshoot/investigate-alert/tools/prometheus-query.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-query.sh
new file mode 100755
index 0000000..76c0ee4
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/prometheus-query.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+# Run an instant PromQL query (current value).
+#
+# Usage: prometheus-query.sh '<promql>'
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+require_arg "${1:-}" "PromQL expression" "Usage: prometheus-query.sh '<promql>'"
+
+QUERY=$(urlencode "$1")
+prometheus_get "/api/v1/query?query=${QUERY}" | jq .
diff --git a/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh b/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh
new file mode 100755
index 0000000..e9badb4
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/prometheus-setup.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+# Establish Prometheus access credentials for the session.
+# Prints export commands to stdout for eval.
+#
+# Usage: eval $(bash tools/prometheus-setup.sh)
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command oc
+require_command jq
+check_cluster_access
+
+TOKEN=$(oc whoami -t 2>/dev/null) || \
+  error_json "AUTH_FAILED" "Cannot get bearer token" "Run: oc login <cluster-url>"
+
+THANOS_URL=$(oc get route thanos-querier -n openshift-monitoring -o jsonpath='{.spec.host}' 2>/dev/null) || \
+  error_json "ROUTE_NOT_FOUND" "thanos-querier route not found in openshift-monitoring" \
+    "Verify the openshift-monitoring namespace exists and the route is exposed"
+
+if [[ -z "$THANOS_URL" ]]; then
+  error_json "ROUTE_NOT_FOUND" "thanos-querier route has no host" \
+    "Check: oc get route thanos-querier -n openshift-monitoring"
+fi
+
+printf 'export TOKEN=%q\n' "$TOKEN"
+printf 'export THANOS_URL=%q\n' "$THANOS_URL"
diff --git a/cluster-troubleshoot/investigate-alert/tools/query-alert.sh b/cluster-troubleshoot/investigate-alert/tools/query-alert.sh
new file mode 100755
index 0000000..8256cfe
--- /dev/null
+++ b/cluster-troubleshoot/investigate-alert/tools/query-alert.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Query Prometheus for firing instances of a specific alert.
+# Returns the full label set for each firing instance.
+#
+# Usage: query-alert.sh <alert_name>
+
+SCRIPT_DIR="$(dirname "$0")"
+. "$SCRIPT_DIR/common.sh"
+
+require_command jq
+require_command python3
+require_arg "${1:-}" "alert name" "Usage: query-alert.sh <alert_name>"
+
+ALERT_NAME="$1"
+if [[ ! "$ALERT_NAME" =~ ^[a-zA-Z_:][a-zA-Z0-9_:]*$ ]]; then
+  error_json "INVALID_ARG" "Invalid alert name: '$ALERT_NAME'" \
+    "Alert names must match ^[a-zA-Z_:][a-zA-Z0-9_:]*$"
+fi
+PROMQL="ALERTS{alertname=\"${ALERT_NAME}\"}"
+QUERY=$(urlencode "$PROMQL")
+
+RESPONSE=$(prometheus_get "/api/v1/query?query=${QUERY}")
+
+echo "$RESPONSE" | jq --arg name "$ALERT_NAME" '{
+  alert_name: $name,
+  firing: (.data.result | length > 0),
+  instance_count: (.data.result | length),
+  instances: .data.result
+}'
diff --git a/evals/skills/investigate-alert/system_prompt.md b/evals/skills/investigate-alert/system_prompt.md
new file mode 100644
index 0000000..2090cd0
--- /dev/null
+++ b/evals/skills/investigate-alert/system_prompt.md
@@ -0,0 +1,18 @@
+You are an alert investigation assistant with access to the 'investigate-alert' skill. This skill provides shell scripts under tools/ for investigating firing OpenShift alerts: determining root cause and recommending remediation options.
+
+Available diagnostic scripts (all under tools/):
+- prometheus-setup.sh — Set TOKEN and THANOS_URL for Prometheus access
+- query-alert.sh <alert_name> — Firing instances of a named alert
+- fetch-alert-rule.sh <alert_name> — Alert PromQL expression, thresholds, annotations
+- prometheus-query.sh '<promql>' — Instant PromQL query
+- prometheus-query-range.sh '<promql>' [duration] [step] — Range query
+- discover-metrics.sh <pattern> — Search metric names by pattern
+- get-firing-alerts.sh [filter] — All firing alerts
+- diagnose-pod.sh <name-or-selector> [namespace] [--logs] — Pod status, conditions, events, logs
+- diagnose-node.sh [node_name] — Node conditions, capacity, taints
+- diagnose-operator.sh [operator_name] — ClusterOperator health
+- diagnose-workload.sh <name> [namespace] [kind] — Deployment/StatefulSet/DaemonSet status
+- get-events.sh [namespace] [minutes] — Events sorted by time
+- check-recent-changes.sh [namespace] [minutes] — Recent rollouts, image pulls
+
+The investigation protocol has two phases: collect evidence first (scope blast radius, set up Prometheus, query alerts, collect workload state and logs, collect events and recent changes, collect all firing alerts), then analyze (correlate alerts, trace causality chains, recommend remediation options).
diff --git a/evals/skills/investigate-alert/test_cases.yaml b/evals/skills/investigate-alert/test_cases.yaml
new file mode 100644
index 0000000..9a5375e
--- /dev/null
+++ b/evals/skills/investigate-alert/test_cases.yaml
@@ -0,0 +1,183 @@
+# investigate-alert skill eval test cases
+#
+# All tests are framed around the skill's core workflow: an alert fires,
+# and the agent follows the investigation protocol to determine root cause.
+#
+# Static tests verify tool-selection and protocol knowledge — given a step
+# in the investigation, the agent must pick the correct diagnostic script.
+#
+# Mock-based tests embed canned tool output and verify the agent
+# interprets diagnostic data correctly in an alert context.
+
+# --- Static tool-selection tests ---
+
+- name: query_firing_alert
+  query: "Alert KubePodCrashLooping is firing. Which investigate-alert diagnostic script should you run to check how many instances are firing and get their full label sets?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"]
+        description: "The script that queries firing instances of a named alert. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "query-alert.sh"
+
+- name: fetch_alert_rule
+  query: "You are investigating alert KubeDeploymentReplicasMismatch. Which investigate-alert diagnostic script retrieves its PromQL expression, thresholds, and annotations?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["query-alert.sh", "fetch-alert-rule.sh", "get-firing-alerts.sh", "prometheus-query.sh", "discover-metrics.sh"]
+        description: "The script that fetches the alert rule definition. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "fetch-alert-rule.sh"
+
+- name: diagnose_pod_from_alert
+  query: "You are investigating alert KubePodCrashLooping. The alert labels show pod web-frontend-7b9d4 in namespace production. Which investigate-alert diagnostic script should you use to inspect this pod's status, conditions, and events?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["diagnose-pod.sh", "diagnose-workload.sh", "diagnose-node.sh", "get-events.sh", "get-firing-alerts.sh"]
+        description: "The script for pod-level diagnostics. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "diagnose-pod.sh"
+
+- name: diagnose_node_from_alert
+  query: "You are investigating alert KubeNodeNotReady. The alert labels point to node worker-3. Which investigate-alert diagnostic script provides node conditions, capacity, taints, and kubelet info?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["diagnose-node.sh", "diagnose-pod.sh", "diagnose-operator.sh", "get-events.sh", "check-recent-changes.sh"]
+        description: "The script for node diagnostics. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "diagnose-node.sh"
+
+- name: diagnose_operator_from_alert
+  query: "You are investigating alert ClusterOperatorDegraded. The alert labels show operator name 'authentication'. Which investigate-alert diagnostic script checks its health, conditions, and versions?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["diagnose-operator.sh", "diagnose-workload.sh", "diagnose-pod.sh", "get-firing-alerts.sh", "check-recent-changes.sh"]
+        description: "The script for ClusterOperator diagnostics. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "diagnose-operator.sh"
+
+- name: check_changes_during_alert
+  query: "You are investigating alert KubeDeploymentReplicasMismatch in namespace checkout. You need to check what changed recently that may have triggered this alert. Which investigate-alert diagnostic script detects recent rollouts, image pulls, and scaling events?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["check-recent-changes.sh", "get-events.sh", "diagnose-workload.sh", "get-firing-alerts.sh", "diagnose-pod.sh"]
+        description: "The script for detecting recent cluster changes. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "check-recent-changes.sh"
+
+- name: prometheus_setup
+  query: "Before querying any alerts with the investigate-alert skill's Prometheus tools, what script must be run first to configure access credentials?"
+  schema:
+    type: object
+    properties:
+      script:
+        type: string
+        enum: ["prometheus-setup.sh", "prometheus-query.sh", "discover-metrics.sh", "get-firing-alerts.sh"]
+        description: "The script that sets up Prometheus credentials. Use the 'investigate-alert' skill to find this."
+    required: [script]
+  expected:
+    script: "prometheus-setup.sh"
+
+- name: first_investigation_step
+  query: "According to the investigate-alert investigation protocol, what is the first command you must run in Step 2 to set up Prometheus access?"
+  schema:
+    type: object
+    properties:
+      command:
+        type: string
+        enum:
+          - "eval $(bash tools/prometheus-setup.sh)"
+          - "bash tools/get-firing-alerts.sh"
+          - "bash tools/diagnose-pod.sh"
+          - "bash tools/get-events.sh"
+        description: "The first command in the investigation protocol. Use the 'investigate-alert' skill to find this."
+    required: [command]
+  expected:
+    command: "eval $(bash tools/prometheus-setup.sh)"
+
+- name: step3_mandatory_logs
+  query: "You are investigating alert KubeDeploymentReplicasMismatch. You have already run diagnose-workload.sh for the affected deployment. According to the investigation protocol Step 3, what must you also run for every workload to collect container logs, even if the workload appears healthy?"
+  schema:
+    type: object
+    properties:
+      tool:
+        type: string
+        enum: ["diagnose-pod.sh --logs", "get-events.sh", "check-recent-changes.sh", "prometheus-query.sh"]
+        description: "The additional evidence to collect per workload in Step 3. Use the 'investigate-alert' skill to find this."
+    required: [tool]
+  expected:
+    tool: "diagnose-pod.sh --logs"
+
+# --- Mock-based diagnostic reasoning tests ---
+
+- name: interpret_firing_alert
+  query: |
+    You are investigating alert KubeDeploymentReplicasMismatch. You ran query-alert.sh and got:
+    {"alert_name":"KubeDeploymentReplicasMismatch","firing":true,"instance_count":2,"instances":[{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"payment-gateway","severity":"warning"},"value":[1721203200,"1"]},{"metric":{"__name__":"ALERTS","alertname":"KubeDeploymentReplicasMismatch","namespace":"checkout","deployment":"order-service","severity":"warning"},"value":[1721203200,"1"]}]}
+    Based on this output: is the alert firing, how many instances are affected, and in which namespace?
+  schema:
+    type: object
+    properties:
+      is_firing:
+        type: boolean
+        description: "Whether the alert is currently firing"
+      instance_count:
+        type: integer
+        description: "Number of firing instances"
+      namespace:
+        type: string
+        enum: ["checkout", "payments", "production", "default", "openshift-monitoring"]
+        description: "The namespace where the alert is firing"
+    required: [is_firing, instance_count, namespace]
+  expected:
+    is_firing: true
+    instance_count: 2
+    namespace: "checkout"
+
+- name: interpret_pod_oomkilled
+  query: |
+    You are investigating alert KubePodCrashLooping. The alert labels pointed to pod api-server-6f8b9c-xk2lm in namespace payments. You ran diagnose-pod.sh and got this output:
+    {"pods":[{"name":"api-server-6f8b9c-xk2lm","namespace":"payments","phase":"Running","conditions":[{"type":"Ready","status":"False"},{"type":"ContainersReady","status":"False"}],"container_statuses":[{"name":"api","state":{"waiting":{"reason":"CrashLoopBackOff"}},"lastState":{"terminated":{"reason":"OOMKilled","exitCode":137,"finishedAt":"2026-07-17T08:15:00Z"}},"restartCount":12,"ready":false}],"node_name":"worker-2","restart_count":12,"events":[{"type":"Warning","reason":"BackOff","message":"Back-off restarting failed container api in pod api-server-6f8b9c-xk2lm_payments"}]}]}
+    What is the root cause of this pod's failure?
+  schema:
+    type: object
+    properties:
+      root_cause:
+        type: string
+        enum: ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff", "Evicted", "NodeNotReady"]
+        description: "The underlying cause of the container's termination."
+      pod_name:
+        type: string
+        description: "The name of the affected pod"
+      restart_count:
+        type: integer
+        description: "Number of container restarts"
+    required: [root_cause, pod_name, restart_count]
+  expected:
+    root_cause: "OOMKilled"
+    pod_name: "api-server-6f8b9c-xk2lm"
+    restart_count: 12