From a1f0e96e451c8b174cc44a9ab4bbb9355b91b86e Mon Sep 17 00:00:00 2001 From: harshit-bstack Date: Wed, 29 Jul 2026 12:18:51 +0530 Subject: [PATCH 01/51] Skill Update --- agents/ai-tfa-coordinator.md | 53 +++++++++++++++++-- skills/rca-build/SKILL.md | 98 ++++++++++++++++++++++++++++++++++-- workflows/rca-batch.mjs | 25 +++++++-- 3 files changed, 167 insertions(+), 9 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 6806b15..c69efcf 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -64,9 +64,42 @@ read-only and has no side effects, so a read is always safe to repeat. ## Operating principles -1. **Logs by TFA — the core contract.** Never seed logs in the first turn; - **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or - digest log content. Logs are TFA's job. +0. **Mandatory connector sweeps — proactive, never ask-gated.** This is about + **server-side app logs** (e.g. `kubectl logs` on the llmservice/ + testcasegeneration pods) — a different thing from the `test_logs` evidence + type in Principle 1 below, which is the test's own execution log (SDK + trace/driver output) and stays exclusively TFA's. There is no conflict + between the two: Principle 1 forbids you from ever fetching `test_logs`; + Principle 0 requires you to fetch **app logs**, a connector-owned capability + distinct from `test_logs`, whenever the connector declares it compulsory. + Before submitting turn 1, check every capability the manifest marks + `available` for a declared compulsory check in its connector skill (read the + skill's `SKILL.md` — look for language like "COMPULSORY", "not conditional", + "not a fallback", "every RCA that touches this skill MUST..."). If a + capability declares one (e.g. `nl2steps-infra`'s "kubectl app-log check is + COMPULSORY"), **run it now, on turn 1, unconditionally** and fold its + evidence block (`KUBECTL_APP_LOGS:` or the skill's equivalent) into the + first-turn message — do **not** wait for a `NEEDS_INFO` ask whose + `evidenceType` names that capability. **TFA's `evidenceType` label is not a + reliable gate for this**: TFA may tag a deploy/infra-shaped question + `product_code` (observed: "was there a recent deploy of `llmservice`?" + tagged `product_code`, never routed to infra) — a compulsory check must run + regardless of how TFA labels its asks, or it never runs at all. Re-run only + once per test (cache the result across turns/asks on the same thread); a + later ask for the same evidenceType is satisfied from the cached sweep, not + re-fetched. **A connector skill may also declare a minimum call-count budget** + (e.g. `nl2steps-infra`'s "at least 5 separate `kubectl` invocations per RCA + turn" — a latency-instrumentation baseline, not busywork: it exists so the + infra/k8s path has enough call volume to measure latency against, matching + the github connector's naturally higher call count). Honor a declared + minimum exactly — never satisfy it with fewer, batched/combined, or cached + calls; each counted call must be a real, individually useful invocation. +1. **Logs by TFA — the core contract.** Never seed **test execution logs** + (`test_logs` evidenceType — SDK trace, driver output, screenshots) in the + first turn; **skip every ask with `evidenceType === "test_logs"`**. Never + fetch, paste, or digest test-log content. That log is TFA's job — this rule + does not cover the app/infra logs Principle 0 requires; those are a + separate, connector-owned evidence type and are never `test_logs`. 2. **Read-only.** Every gather mechanism is read-only. Never write to a repo, cluster, ticket, or the run. Produce a block and stop. 3. **Turn-cap** = `turnCap` from `config/rca.config.json` (default 6). If the cap @@ -136,6 +169,12 @@ re-fetch per test. Never fabricate a PR when the github capability is unavailabl Suspect PR(s): . Confirm against THIS test's logs." (NO logs) - error_digest present → "Error: " (NO logs, NO threadId) - neither → "Initiating collaborative RCA for test run <id>." +0.5. MANDATORY CONNECTOR SWEEPS (Operating Principle 0): for every `available` + capability, check its connector skill for a declared compulsory check. Run + any that apply NOW — before turn 1, regardless of pre_seed/error_digest + content — and append each one's evidence block to the DIGEST. Record what + ran in `mandatory_checks` for the final RCA_OUTPUT. This step runs exactly + once per test (cache across turns); do not re-run on a later matching ask. 1. SUBMIT turn 1: tfaRcaTurn(testRunId=<id>, message=<digest>). Capture threadId. turns_used = 1. (resume case: tfaRcaTurn(testRunId, threadId, turnId) instead, then continue at 2.) 2. CLASSIFY result.status: @@ -218,6 +257,14 @@ RCA_OUTPUT_START ## turns_used <integer 1..turnCap> +## mandatory_checks +- <capability>: ran (<M> calls) — <one-line evidence summary, e.g. "kubectl: ran (5 calls) — clean (window=t±2m, 2 pods)"> +- <capability>: ran (<M> calls) — <N matched lines — one-line digest> +- <capability>: not-applicable — <capability declares no compulsory check> +"none" only if no available capability declares any compulsory check. If the +connector declares a minimum call-count budget, `<M>` must be >= that minimum — +report the actual count run, not the minimum itself, so a shortfall is visible. + ## asks_fulfilled - <evidenceType> # every non-test_logs type fulfilled; "none" if empty diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index e979813..e63e623 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -44,7 +44,53 @@ pass. The gate has two parts; both run before any RCA work starts. ### Part A — connector discovery + validation -Enumerate every connector relevant to test RCA: +**Step 0 — enumerate connector-shaped skills FIRST (before probing raw MCP tools).** +Run: + +```bash +ls .claude/skills/ ~/.claude/skills/ 2>/dev/null +``` + +For each `SKILL.md` found, open it and look for a **Capability declaration** +block (or a `capability: <name>` line in the frontmatter/body). Any skill that +declares `capability: github | infra | logs | metrics | other` **IS** the +connector for that capability and MUST be added to the manifest — it +**SUPERSEDES** the raw MCP tool for that capability because it carries +product-specific routing (repo map, cluster/namespace, branch conventions, +falsification protocol) the raw tool does not. Record the skill name in the +manifest entry (e.g. `github: valid, via: gh (skill=nl2steps-github)`). Skipping +this step is the failure mode where the orchestrator dispatches coordinators +that grep the wrong repos on the wrong branch. + +**Disambiguating by product (nudge / one-question rule).** Connector skills are +product-scoped — a workspace may hold none, one, or several product families +(e.g. `nl2steps-*`, `o11y-*`, `tcm-*`, whatever the user has). After the `ls`, +pick the *product family* whose connector skills apply to THIS build: + +- **Zero families found** → **nudge the user in the gate summary**: + "No connector-shaped skills found under `.claude/skills/` — proceeding with + raw MCP tools only; culprit-PR attribution will be best-effort against + workspace `git remote` guesses. Add a `<product>-github` / `<product>-infra` + skill for higher-fidelity routing." Then proceed with raw connectors. **Do + NOT block.** +- **Exactly one family** → use it. No question. +- **Multiple families** (`nl2steps-*` AND `o11y-*` AND `tcm-*` …) → try to + disambiguate WITHOUT asking: + 1. Match the build's project / build name (from `getBuildId` metadata or + the invocation args) against each family's SKILL.md description / product + hints — if one family matches unambiguously, use it. + 2. Match the discovered failure signatures (from Step 2's `listTestIds` if it + has already run, else defer this to a re-visit after discovery) against + each family's declared file paths / error patterns — if one family owns + the failure surface, use it. + If both signals leave the choice ambiguous, this earns the **one + consolidated gate question** (Part B rules apply): fold it into the same + question as any other non-assumable field, e.g. *"Multiple product families + found (`nl2steps`, `o11y`, `tcm`); build/failure signatures don't uniquely + pick one — which family owns this build's failures?"* Headless: pick the + first alphabetically and record the ambiguity as a gap. + +Then enumerate every connector relevant to test RCA: - from `config/rca.config.json` → `evidenceRouting`: **github** (product_code/deploy/ci), **infra** (whatever runtime the user has — k8s, @@ -63,6 +109,31 @@ Enumerate every connector relevant to test RCA: | metrics | a metrics skill/MCP tool actually listed in the session | | other | best-effort; default `absent` | +**Scope validation — run the SCOPE PROBES declared by each connector skill.** +The base probe above (`gh auth status`, `kubectl version`, …) only confirms the +raw tool works. It does not confirm the *concrete targets* a coordinator will +touch — specific repos, branches, clusters, namespaces, indices — are actually +reachable. That is the connector SKILL's job: each connector skill MUST +declare, in its `Capability declaration` section, a `Scope probes:` list +naming what to check and how. This orchestrator's contract is generic: + +1. For every connector skill added to the manifest in Step 0, read its + `Scope probes:` list. +2. Run each probe verbatim. +3. Record every target's result in the manifest entry — passes go into a + resolved-scope field (e.g. `repos_validated: [...]`, `namespace: ok`), + failures go into a per-target gap (e.g. `<target>: 404 not_accessible`). +4. A per-target failure is a scoped gap, not a connector-wide failure — the + connector stays `valid` for the targets that did pass. + +Coordinators can then act freely inside the resolved scope and must fail +closed outside it. This closes the failure mode where a coordinator degrades +to `unavailable` because the orchestrator didn't confirm the specific target. + +Skills that don't declare `Scope probes:` degrade to a manifest-time warning +("scope probes missing — coordinator may over-degrade"). Do not invent +product-specific probes here. + Output the **validated capability manifest**: `connector → valid | invalid | absent` (`lib/routing.mjs` → `buildManifest`; `valid` maps to `available: true`). An `invalid` or `absent` connector is a **recorded gap** — @@ -167,9 +238,14 @@ reuse it, do not re-discover): ## Step 5 — fan-out (fully autonomous) -Drive the cluster work-list, **`concurrency` (default 5) at a time**: +Drive the cluster work-list, **`concurrency` (default 50) at a time**: representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL -(claim → heartbeat → flip) so the run is resumable. +(claim → heartbeat → flip) so the run is resumable. On the Claude Code / +Workflow-tool path, this is a soft target only — the Workflow runtime hard-caps +actual concurrent `agent()` calls at `min(16, cpu cores - 2)` regardless of +this config value; excess work queues and runs as slots free up rather than +running 50-wide. The sequential harness / manual subagent-dispatch path has no +such ceiling and will honor `concurrency` literally. - Claude Code → run the dynamic workflow `workflows/rca-batch.mjs` (script-orchestrated; gap → "unavailable" back to TFA → best-effort finalize). @@ -182,6 +258,15 @@ coordinator that dies becomes a recorded `failed` row — one stuck test never sinks the batch (partial-first). No path ever prompts the user (the gate is closed). +**Coordinator prompts MUST name every connector-shaped skill on the manifest.** +Each dispatch prompt lists, per capability, the resolved connector skill from +Gate Part A Step 0 — e.g. *"Use `nl2steps-github` for every product_code / +deploy / ci ask (canonical repos + branch live in the skill; do NOT grep other +repos). Use `nl2steps-infra` for every infra ask."* A coordinator prompt that +omits a manifest-listed connector skill — and that therefore lets the +coordinator infer repos from workspace `git remote` or cwd — is a bug: the +coordinator will land plausible-but-wrong PR attributions on adjacent repos. + **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure @@ -245,3 +330,10 @@ thread. the Test Observability UI link only. - A PRODUCT_BUG RCA without a GitHub PR link is incomplete — dig until the turn cap, else state what was searched and record the gap. +- A connector skill's own compulsory mandate (e.g. `nl2steps-infra`'s "kubectl + app-log check is COMPULSORY") is honored **proactively on turn 1** — never + gated on TFA naming that evidenceType in an ask. TFA is observed to mislabel + deploy/infra-shaped questions as `product_code`, so ask-routing alone cannot + be trusted to trigger a compulsory check; the coordinator runs it unconditionally + (`agents/ai-tfa-coordinator.md` Operating Principle 0) and records it under + `mandatory_checks` in the RCA_OUTPUT. diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index f79e7a2..806e344 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -49,6 +49,7 @@ const RCA_SCHEMA = { asks_skipped: { type: "array", items: { type: "string" } }, asks_unavailable: { type: "array", items: { type: "string" } }, cluster_id: { type: "string" }, + mandatory_checks: { type: "array", items: { type: "string" } }, }, additionalProperties: true, }; @@ -63,8 +64,23 @@ const shared = [ `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, `Persist eagerly to the CSV: claim your row before turn 1, flip it on terminal (lib/csv-state.mjs).`, + `MANDATORY CONNECTOR SWEEPS (Operating Principle 0, ai-tfa-coordinator.md): before turn 1, check every available capability's connector skill for a declared compulsory check (e.g. nl2steps-infra's "kubectl app-log check is COMPULSORY — not conditional, not a fallback"). Run any that apply NOW, unconditionally, and fold the evidence block into the turn-1 message. Do NOT wait for a NEEDS_INFO ask naming that evidenceType — TFA has been observed to label deploy/infra-shaped questions "product_code", so ask-routing alone will never trigger it. Record what ran under mandatory_checks in the RCA_OUTPUT.`, + `MINIMUM CALL BUDGET: nl2steps-infra requires AT LEAST 5 separate real kubectl invocations per RCA turn that touches it (deploy state, pod-discovery x2 as SEPARATE calls, log-sweep x2 minimum — see the skill's "Minimum call budget" section). This is a latency-instrumentation baseline so the k8s path has comparable call volume to the github connector, not busywork — never satisfy it by combining calls, batching selectors, or reusing a cached result. Report the actual count run in mandatory_checks (e.g. "kubectl: ran (5 calls) — ...").`, ].join("\n"); +function resumeLine(row) { + if (!row?.threadId || !row?.turnId) return null; + return [ + `RESUME (do not start a new thread): this test already has an in-flight thread`, + `threadId=${row.threadId} turnId=${row.turnId}.`, + `Call getTfaTurnResult(testRunId, turnId) FIRST to read its current state`, + `(drain any soft-PENDING per the softPendingDrain budget) before submitting`, + `anything further — reuse this threadId for every follow-up on this test.`, + row.last_evidence_digest ? `Prior evidence already gathered (reuse, don't re-fetch): ${row.last_evidence_digest}` : null, + row.root_cause ? `Prior attempt note: ${row.root_cause}` : null, + ].filter(Boolean).join("\n"); +} + function repPrompt(cluster) { const r = cluster.representative; return [ @@ -72,9 +88,10 @@ function repPrompt(cluster) { `Run the FULL collaborative RCA loop for the representative test.`, `testRunId=${r.testRunId} testName=${r.testName ?? ""}`, `error_digest: ${r.error_summary ?? "(none)"}`, + resumeLine(r), shared, `Return the structured RCA_OUTPUT for this test.`, - ].join("\n"); + ].filter(Boolean).join("\n"); } function siblingPrompt(sibling, repResult, cluster) { @@ -87,9 +104,10 @@ function siblingPrompt(sibling, repResult, cluster) { `If TFA confirms in one turn → done. If it does NOT (NEEDS_INFO), fall back to the full loop — never blindly inherit.`, `testRunId=${sibling.testRunId} testName=${sibling.testName ?? ""}`, `error_digest: ${sibling.error_summary ?? "(none)"}`, + resumeLine(sibling), shared, `Return the structured RCA_OUTPUT for this test.`, - ].join("\n"); + ].filter(Boolean).join("\n"); } log(`Batch: ${clusters.length} cluster(s) over build ${ctx.buildId ?? "?"}`); @@ -97,7 +115,8 @@ log(`Batch: ${clusters.length} cluster(s) over build ${ctx.buildId ?? "?"}`); // Pipeline: each cluster flows representative → siblings independently (no barrier // between stages), so a small cluster's siblings confirm while a big cluster's // representative is still looping. Concurrency is bounded by the workflow runtime -// (~min(16, cores-2)); config.concurrency (5) is the intended soft target. +// (~min(16, cores-2)) regardless of config.concurrency (50) — that value is the +// intended soft target/upper bound; the runtime queues anything beyond its own cap. const results = await pipeline( clusters, (cluster) => From 4429fbae5127622d07d8388aa57cee76a1c193d6 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Wed, 29 Jul 2026 17:46:23 +0530 Subject: [PATCH 02/51] Update --- config/rca.config.json | 3 ++- skills/rca-build/SKILL.md | 42 +++++++++++++++++++++++++++------------ workflows/rca-batch.mjs | 9 ++++++--- 3 files changed, 37 insertions(+), 17 deletions(-) diff --git a/config/rca.config.json b/config/rca.config.json index 802f6f4..45c3997 100644 --- a/config/rca.config.json +++ b/config/rca.config.json @@ -1,7 +1,8 @@ { "$comment": "Central config for the /rca-build RCA harness. All formerly-hardcoded product/infra values live here. No kubectl/chitragupta/bifrost literals — connectors are discovered and probe-validated at the gate into the capability manifest (see skills/rca-build/references/evidence-routing.md). No reportFile: the plugin never writes a local RCA report — the full report lives on the Test Observability UI (triggerRcaReport).", "mcpServerName": "bstack", - "concurrency": 5, + "$concurrencyComment": "Max coordinator subagents run in parallel during Step 5 fan-out. HONORED LITERALLY on the default path (direct Agent-tool dispatch of ai-tfa-coordinator subagents — one message, up to `concurrency` tool-use blocks per batch) and by the sequential harness (lib/loop.mjs). Only a SOFT target on the opt-in Workflow-tool path (workflows/rca-batch.mjs), which the Workflow runtime hard-caps at min(16, cpu cores - 2) regardless of this value — that cap is architectural and cannot be raised from this repo. See SKILL.md Step 5.", + "concurrency": 20, "turnCap": 6, "turnMessageMaxChars": 5000, "pollSoftPendingMs": 90000, diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index e63e623..d802293 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -238,20 +238,36 @@ reuse it, do not re-discover): ## Step 5 — fan-out (fully autonomous) -Drive the cluster work-list, **`concurrency` (default 50) at a time**: +Drive the cluster work-list, **`concurrency` (default 20) at a time**: representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL -(claim → heartbeat → flip) so the run is resumable. On the Claude Code / -Workflow-tool path, this is a soft target only — the Workflow runtime hard-caps -actual concurrent `agent()` calls at `min(16, cpu cores - 2)` regardless of -this config value; excess work queues and runs as slots free up rather than -running 50-wide. The sequential harness / manual subagent-dispatch path has no -such ceiling and will honor `concurrency` literally. - -- Claude Code → run the dynamic workflow `workflows/rca-batch.mjs` - (script-orchestrated; gap → "unavailable" back to TFA → best-effort finalize). -- Hosts without the Workflow runtime → dispatch `tfa-rca:ai-tfa-coordinator` - subagents ≤ `concurrency` at a time, or drive the sequential harness - `lib/loop.mjs` (`runRcaLoop`). Same contract, same no-prompt rule. +(claim → heartbeat → flip) so the run is resumable. + +> **Concurrency comes from `config/rca.config.json` — always read it from +> there, never hardcode.** The default path (direct Agent-tool dispatch) honors +> the JSON value literally: fan out coordinator subagents in batches of +> `concurrency` (one message, up to `concurrency` tool-use blocks per batch). +> The opt-in `workflows/rca-batch.mjs` path is subject to the Workflow tool's +> architectural cap of `min(16, cpu cores - 2)` — on that path `concurrency` +> is a soft upper bound and excess work queues rather than running N-wide. +> If you need literal fan-out, use the default direct-dispatch path. + +- **Default (all hosts, including Claude Code) → direct Agent-tool dispatch.** + Read `concurrency` from `config/rca.config.json` and dispatch + `tfa-rca:ai-tfa-coordinator` subagents in batches of that size (one message, + up to `concurrency` tool-use blocks per batch). This path is **outside the + Workflow runtime**, so the `min(16, cores-2)` ceiling does not apply and the + JSON value is honored literally. Prefer this path whenever the machine's + Workflow cap (`min(16, cores-2)`) would be smaller than the configured + `concurrency` — e.g. an 8-core Mac caps Workflow at 6 while the JSON asks + for 50. +- Opt-in `workflows/rca-batch.mjs` (Claude Code only) → use only when the + Workflow tool's structured `pipeline()`/`parallel()` orchestration, + `resumeFromRunId` resumability, or progress UI is worth the concurrency + trade. On this path `concurrency` is a soft target only — the runtime hard- + caps at `min(16, cores-2)` regardless of the JSON value. +- Hosts without the Workflow runtime and without Agent-tool fan-out → drive + the sequential harness `lib/loop.mjs` (`runRcaLoop`) one test at a time. + Same contract, same no-prompt rule. Subagents/coordinators return compact `RCA_OUTPUT` blocks, never transcripts. A coordinator that dies becomes a recorded `failed` row — one stuck test never diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 806e344..5bcf506 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -66,6 +66,7 @@ const shared = [ `Persist eagerly to the CSV: claim your row before turn 1, flip it on terminal (lib/csv-state.mjs).`, `MANDATORY CONNECTOR SWEEPS (Operating Principle 0, ai-tfa-coordinator.md): before turn 1, check every available capability's connector skill for a declared compulsory check (e.g. nl2steps-infra's "kubectl app-log check is COMPULSORY — not conditional, not a fallback"). Run any that apply NOW, unconditionally, and fold the evidence block into the turn-1 message. Do NOT wait for a NEEDS_INFO ask naming that evidenceType — TFA has been observed to label deploy/infra-shaped questions "product_code", so ask-routing alone will never trigger it. Record what ran under mandatory_checks in the RCA_OUTPUT.`, `MINIMUM CALL BUDGET: nl2steps-infra requires AT LEAST 5 separate real kubectl invocations per RCA turn that touches it (deploy state, pod-discovery x2 as SEPARATE calls, log-sweep x2 minimum — see the skill's "Minimum call budget" section). This is a latency-instrumentation baseline so the k8s path has comparable call volume to the github connector, not busywork — never satisfy it by combining calls, batching selectors, or reusing a cached result. Report the actual count run in mandatory_checks (e.g. "kubectl: ran (5 calls) — ...").`, + `VICTORIALOGS COMPULSORY CHECK: whenever the grafana MCP is present in the session, nl2steps-infra ALSO requires a VictoriaLogs sweep via mcp__grafana__query_loki_logs (CG Non Prod datasource, uid bfq0jp1cji5moc) on every RCA turn that touches this skill — see the skill's "VictoriaLogs check is ALSO COMPULSORY" section. This runs ALONGSIDE the kubectl sweep, never instead of it. Emit a VICTORIALOGS: block (<N> matched lines / clean / not-onboarded / unavailable) — "clean" requires both a real query_loki_logs call AND a confirmed non-zero coverage check this turn; an unconfirmed/zero-coverage product line reports "not-onboarded", not "clean". If the grafana MCP is absent, "VICTORIALOGS: unavailable — no grafana connector this run" is sufficient. Report what ran under mandatory_checks, distinct from the kubectl entry.`, ].join("\n"); function resumeLine(row) { @@ -114,9 +115,11 @@ log(`Batch: ${clusters.length} cluster(s) over build ${ctx.buildId ?? "?"}`); // Pipeline: each cluster flows representative → siblings independently (no barrier // between stages), so a small cluster's siblings confirm while a big cluster's -// representative is still looping. Concurrency is bounded by the workflow runtime -// (~min(16, cores-2)) regardless of config.concurrency (50) — that value is the -// intended soft target/upper bound; the runtime queues anything beyond its own cap. +// representative is still looping. Concurrency is capped by the Workflow runtime +// at min(16, cores-2) — an architectural limit of the tool itself, not something +// this script or config.concurrency (20, see rca.config.json) can raise. That +// config value is an intended soft target/upper bound on THIS path only; the +// runtime queues anything beyond its own cap regardless of what this file says. const results = await pipeline( clusters, (cluster) => From b733759bf30301f548b362d60891551dbec79ece Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Thu, 30 Jul 2026 16:23:37 +0530 Subject: [PATCH 03/51] fix(rca): revert testing-only hardcoding from v2 latency study MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Strips product-specific artifacts that leaked into the generic plugin from the nl2steps latency-instrumentation runs: the hardcoded "at least 5 kubectl calls" busywork floor and the hardcoded VictoriaLogs Grafana datasource UID in the shared coordinator prompt, plus de-hardcoded remaining nl2steps/o11y/tcm example references in SKILL.md and ai-tfa-coordinator.md down to neutral placeholders. Restores concurrency to 20 (the profiling-backed fleet ceiling) instead of the untested 50. Also gitignores .DS_Store/*.code-workspace to keep local editor artifacts out of the repo. The generic "mandatory connector sweep" mechanism itself (a connector skill may declare a compulsory check, routed by capability) is kept — only the instrumentation-specific numbers/UID were testing artifacts. --- .gitignore | 2 ++ agents/ai-tfa-coordinator.md | 28 ++++++++++------------------ skills/rca-build/SKILL.md | 27 ++++++++++++++------------- workflows/rca-batch.mjs | 4 +--- 4 files changed, 27 insertions(+), 34 deletions(-) diff --git a/.gitignore b/.gitignore index 4c14fb3..433d1ce 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,7 @@ node_modules/ .env +.DS_Store +*.code-workspace # Per-run RCA batch state (the CSV/WAL spine + report) is workspace-local. .rca/ # Planning docs (brainstorm/ideation/plan) stay local — not pushed. diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index c69efcf..a93e9b3 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -76,24 +76,16 @@ read-only and has no side effects, so a read is always safe to repeat. `available` for a declared compulsory check in its connector skill (read the skill's `SKILL.md` — look for language like "COMPULSORY", "not conditional", "not a fallback", "every RCA that touches this skill MUST..."). If a - capability declares one (e.g. `nl2steps-infra`'s "kubectl app-log check is - COMPULSORY"), **run it now, on turn 1, unconditionally** and fold its - evidence block (`KUBECTL_APP_LOGS:` or the skill's equivalent) into the - first-turn message — do **not** wait for a `NEEDS_INFO` ask whose - `evidenceType` names that capability. **TFA's `evidenceType` label is not a - reliable gate for this**: TFA may tag a deploy/infra-shaped question - `product_code` (observed: "was there a recent deploy of `llmservice`?" - tagged `product_code`, never routed to infra) — a compulsory check must run - regardless of how TFA labels its asks, or it never runs at all. Re-run only - once per test (cache the result across turns/asks on the same thread); a - later ask for the same evidenceType is satisfied from the cached sweep, not - re-fetched. **A connector skill may also declare a minimum call-count budget** - (e.g. `nl2steps-infra`'s "at least 5 separate `kubectl` invocations per RCA - turn" — a latency-instrumentation baseline, not busywork: it exists so the - infra/k8s path has enough call volume to measure latency against, matching - the github connector's naturally higher call count). Honor a declared - minimum exactly — never satisfy it with fewer, batched/combined, or cached - calls; each counted call must be a real, individually useful invocation. + capability declares one, **run it now, on turn 1, unconditionally** and fold + its evidence block into the first-turn message — do **not** wait for a + `NEEDS_INFO` ask whose `evidenceType` names that capability. **TFA's + `evidenceType` label is not a reliable gate for this**: TFA may tag a + deploy/infra-shaped question `product_code` (observed: "was there a recent + deploy of `llmservice`?" tagged `product_code`, never routed to infra) — a + compulsory check must run regardless of how TFA labels its asks, or it + never runs at all. Re-run only once per test (cache the result across + turns/asks on the same thread); a later ask for the same evidenceType is + satisfied from the cached sweep, not re-fetched. 1. **Logs by TFA — the core contract.** Never seed **test execution logs** (`test_logs` evidenceType — SDK trace, driver output, screenshots) in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index d802293..2966bcc 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -58,13 +58,13 @@ connector for that capability and MUST be added to the manifest — it **SUPERSEDES** the raw MCP tool for that capability because it carries product-specific routing (repo map, cluster/namespace, branch conventions, falsification protocol) the raw tool does not. Record the skill name in the -manifest entry (e.g. `github: valid, via: gh (skill=nl2steps-github)`). Skipping +manifest entry (e.g. `github: valid, via: gh (skill=<product>-github)`). Skipping this step is the failure mode where the orchestrator dispatches coordinators that grep the wrong repos on the wrong branch. **Disambiguating by product (nudge / one-question rule).** Connector skills are product-scoped — a workspace may hold none, one, or several product families -(e.g. `nl2steps-*`, `o11y-*`, `tcm-*`, whatever the user has). After the `ls`, +(e.g. `<product-a>-*`, `<product-b>-*`, whatever the user has). After the `ls`, pick the *product family* whose connector skills apply to THIS build: - **Zero families found** → **nudge the user in the gate summary**: @@ -74,7 +74,7 @@ pick the *product family* whose connector skills apply to THIS build: skill for higher-fidelity routing." Then proceed with raw connectors. **Do NOT block.** - **Exactly one family** → use it. No question. -- **Multiple families** (`nl2steps-*` AND `o11y-*` AND `tcm-*` …) → try to +- **Multiple families** (e.g. `<product-a>-*` AND `<product-b>-*` …) → try to disambiguate WITHOUT asking: 1. Match the build's project / build name (from `getBuildId` metadata or the invocation args) against each family's SKILL.md description / product @@ -86,9 +86,9 @@ pick the *product family* whose connector skills apply to THIS build: If both signals leave the choice ambiguous, this earns the **one consolidated gate question** (Part B rules apply): fold it into the same question as any other non-assumable field, e.g. *"Multiple product families - found (`nl2steps`, `o11y`, `tcm`); build/failure signatures don't uniquely - pick one — which family owns this build's failures?"* Headless: pick the - first alphabetically and record the ambiguity as a gap. + found (`<product-a>`, `<product-b>`); build/failure signatures don't + uniquely pick one — which family owns this build's failures?"* Headless: + pick the first alphabetically and record the ambiguity as a gap. Then enumerate every connector relevant to test RCA: @@ -259,7 +259,7 @@ representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL JSON value is honored literally. Prefer this path whenever the machine's Workflow cap (`min(16, cores-2)`) would be smaller than the configured `concurrency` — e.g. an 8-core Mac caps Workflow at 6 while the JSON asks - for 50. + for 20. - Opt-in `workflows/rca-batch.mjs` (Claude Code only) → use only when the Workflow tool's structured `pipeline()`/`parallel()` orchestration, `resumeFromRunId` resumability, or progress UI is worth the concurrency @@ -276,10 +276,11 @@ closed). **Coordinator prompts MUST name every connector-shaped skill on the manifest.** Each dispatch prompt lists, per capability, the resolved connector skill from -Gate Part A Step 0 — e.g. *"Use `nl2steps-github` for every product_code / -deploy / ci ask (canonical repos + branch live in the skill; do NOT grep other -repos). Use `nl2steps-infra` for every infra ask."* A coordinator prompt that -omits a manifest-listed connector skill — and that therefore lets the +Gate Part A Step 0 — e.g. *"Use `<resolved-github-skill>` for every +product_code / deploy / ci ask (canonical repos + branch live in the skill; do +NOT grep other repos). Use `<resolved-infra-skill>` for every infra ask."* A +coordinator prompt that omits a manifest-listed connector skill — and that +therefore lets the coordinator infer repos from workspace `git remote` or cwd — is a bug: the coordinator will land plausible-but-wrong PR attributions on adjacent repos. @@ -346,8 +347,8 @@ thread. the Test Observability UI link only. - A PRODUCT_BUG RCA without a GitHub PR link is incomplete — dig until the turn cap, else state what was searched and record the gap. -- A connector skill's own compulsory mandate (e.g. `nl2steps-infra`'s "kubectl - app-log check is COMPULSORY") is honored **proactively on turn 1** — never +- A connector skill's own compulsory mandate (e.g. an infra connector marking + its app-log check "COMPULSORY") is honored **proactively on turn 1** — never gated on TFA naming that evidenceType in an ask. TFA is observed to mislabel deploy/infra-shaped questions as `product_code`, so ask-routing alone cannot be trusted to trigger a compulsory check; the coordinator runs it unconditionally diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 5bcf506..925b5ac 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -64,9 +64,7 @@ const shared = [ `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, `Persist eagerly to the CSV: claim your row before turn 1, flip it on terminal (lib/csv-state.mjs).`, - `MANDATORY CONNECTOR SWEEPS (Operating Principle 0, ai-tfa-coordinator.md): before turn 1, check every available capability's connector skill for a declared compulsory check (e.g. nl2steps-infra's "kubectl app-log check is COMPULSORY — not conditional, not a fallback"). Run any that apply NOW, unconditionally, and fold the evidence block into the turn-1 message. Do NOT wait for a NEEDS_INFO ask naming that evidenceType — TFA has been observed to label deploy/infra-shaped questions "product_code", so ask-routing alone will never trigger it. Record what ran under mandatory_checks in the RCA_OUTPUT.`, - `MINIMUM CALL BUDGET: nl2steps-infra requires AT LEAST 5 separate real kubectl invocations per RCA turn that touches it (deploy state, pod-discovery x2 as SEPARATE calls, log-sweep x2 minimum — see the skill's "Minimum call budget" section). This is a latency-instrumentation baseline so the k8s path has comparable call volume to the github connector, not busywork — never satisfy it by combining calls, batching selectors, or reusing a cached result. Report the actual count run in mandatory_checks (e.g. "kubectl: ran (5 calls) — ...").`, - `VICTORIALOGS COMPULSORY CHECK: whenever the grafana MCP is present in the session, nl2steps-infra ALSO requires a VictoriaLogs sweep via mcp__grafana__query_loki_logs (CG Non Prod datasource, uid bfq0jp1cji5moc) on every RCA turn that touches this skill — see the skill's "VictoriaLogs check is ALSO COMPULSORY" section. This runs ALONGSIDE the kubectl sweep, never instead of it. Emit a VICTORIALOGS: block (<N> matched lines / clean / not-onboarded / unavailable) — "clean" requires both a real query_loki_logs call AND a confirmed non-zero coverage check this turn; an unconfirmed/zero-coverage product line reports "not-onboarded", not "clean". If the grafana MCP is absent, "VICTORIALOGS: unavailable — no grafana connector this run" is sufficient. Report what ran under mandatory_checks, distinct from the kubectl entry.`, + `MANDATORY CONNECTOR SWEEPS (Operating Principle 0, ai-tfa-coordinator.md): before turn 1, check every available capability's connector skill for a declared compulsory check (a connector skill may mark a check "COMPULSORY — not conditional, not a fallback"). Run any that apply NOW, unconditionally, and fold the evidence block into the turn-1 message. Do NOT wait for a NEEDS_INFO ask naming that evidenceType — TFA has been observed to label deploy/infra-shaped questions "product_code", so ask-routing alone will never trigger it. Record what ran under mandatory_checks in the RCA_OUTPUT.`, ].join("\n"); function resumeLine(row) { From 45a1ba88f15ea7a9bac2076f4e1010ca5e5a5b9d Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Thu, 30 Jul 2026 16:32:59 +0530 Subject: [PATCH 04/51] fix(rca): drop the mandatory pre-turn-1 connector sweep entirely Removes the "proactive, never ask-gated" mandatory connector sweep mechanism (Operating Principle 0 + loop step 0.5 in ai-tfa-coordinator.md, the shared-prompt reminder in rca-batch.mjs, and the mandatory_checks field in the RCA_OUTPUT schema/contract and SKILL.md's closing bullet). Forcing evidence gathering into turn 1 regardless of whether TFA actually asked for it bloats every coordinator's first message and burns tokens for evidence that may not be needed. Evidence gathering is now purely reactive to TFA's own NEEDS_INFO asks, routed by capability as before. --- agents/ai-tfa-coordinator.md | 45 +++--------------------------------- skills/rca-build/SKILL.md | 7 ------ workflows/rca-batch.mjs | 2 -- 3 files changed, 3 insertions(+), 51 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index a93e9b3..43f12b7 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -64,34 +64,9 @@ read-only and has no side effects, so a read is always safe to repeat. ## Operating principles -0. **Mandatory connector sweeps — proactive, never ask-gated.** This is about - **server-side app logs** (e.g. `kubectl logs` on the llmservice/ - testcasegeneration pods) — a different thing from the `test_logs` evidence - type in Principle 1 below, which is the test's own execution log (SDK - trace/driver output) and stays exclusively TFA's. There is no conflict - between the two: Principle 1 forbids you from ever fetching `test_logs`; - Principle 0 requires you to fetch **app logs**, a connector-owned capability - distinct from `test_logs`, whenever the connector declares it compulsory. - Before submitting turn 1, check every capability the manifest marks - `available` for a declared compulsory check in its connector skill (read the - skill's `SKILL.md` — look for language like "COMPULSORY", "not conditional", - "not a fallback", "every RCA that touches this skill MUST..."). If a - capability declares one, **run it now, on turn 1, unconditionally** and fold - its evidence block into the first-turn message — do **not** wait for a - `NEEDS_INFO` ask whose `evidenceType` names that capability. **TFA's - `evidenceType` label is not a reliable gate for this**: TFA may tag a - deploy/infra-shaped question `product_code` (observed: "was there a recent - deploy of `llmservice`?" tagged `product_code`, never routed to infra) — a - compulsory check must run regardless of how TFA labels its asks, or it - never runs at all. Re-run only once per test (cache the result across - turns/asks on the same thread); a later ask for the same evidenceType is - satisfied from the cached sweep, not re-fetched. -1. **Logs by TFA — the core contract.** Never seed **test execution logs** - (`test_logs` evidenceType — SDK trace, driver output, screenshots) in the - first turn; **skip every ask with `evidenceType === "test_logs"`**. Never - fetch, paste, or digest test-log content. That log is TFA's job — this rule - does not cover the app/infra logs Principle 0 requires; those are a - separate, connector-owned evidence type and are never `test_logs`. +1. **Logs by TFA — the core contract.** Never seed logs in the first turn; + **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, + or digest log content. Logs are TFA's job. 2. **Read-only.** Every gather mechanism is read-only. Never write to a repo, cluster, ticket, or the run. Produce a block and stop. 3. **Turn-cap** = `turnCap` from `config/rca.config.json` (default 6). If the cap @@ -161,12 +136,6 @@ re-fetch per test. Never fabricate a PR when the github capability is unavailabl Suspect PR(s): <related_prs>. Confirm against THIS test's logs." (NO logs) - error_digest present → "Error: <title + endpoint>" (NO logs, NO threadId) - neither → "Initiating collaborative RCA for test run <id>." -0.5. MANDATORY CONNECTOR SWEEPS (Operating Principle 0): for every `available` - capability, check its connector skill for a declared compulsory check. Run - any that apply NOW — before turn 1, regardless of pre_seed/error_digest - content — and append each one's evidence block to the DIGEST. Record what - ran in `mandatory_checks` for the final RCA_OUTPUT. This step runs exactly - once per test (cache across turns); do not re-run on a later matching ask. 1. SUBMIT turn 1: tfaRcaTurn(testRunId=<id>, message=<digest>). Capture threadId. turns_used = 1. (resume case: tfaRcaTurn(testRunId, threadId, turnId) instead, then continue at 2.) 2. CLASSIFY result.status: @@ -249,14 +218,6 @@ RCA_OUTPUT_START ## turns_used <integer 1..turnCap> -## mandatory_checks -- <capability>: ran (<M> calls) — <one-line evidence summary, e.g. "kubectl: ran (5 calls) — clean (window=t±2m, 2 pods)"> -- <capability>: ran (<M> calls) — <N matched lines — one-line digest> -- <capability>: not-applicable — <capability declares no compulsory check> -"none" only if no available capability declares any compulsory check. If the -connector declares a minimum call-count budget, `<M>` must be >= that minimum — -report the actual count run, not the minimum itself, so a shortfall is visible. - ## asks_fulfilled - <evidenceType> # every non-test_logs type fulfilled; "none" if empty diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 2966bcc..581e834 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -347,10 +347,3 @@ thread. the Test Observability UI link only. - A PRODUCT_BUG RCA without a GitHub PR link is incomplete — dig until the turn cap, else state what was searched and record the gap. -- A connector skill's own compulsory mandate (e.g. an infra connector marking - its app-log check "COMPULSORY") is honored **proactively on turn 1** — never - gated on TFA naming that evidenceType in an ask. TFA is observed to mislabel - deploy/infra-shaped questions as `product_code`, so ask-routing alone cannot - be trusted to trigger a compulsory check; the coordinator runs it unconditionally - (`agents/ai-tfa-coordinator.md` Operating Principle 0) and records it under - `mandatory_checks` in the RCA_OUTPUT. diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 925b5ac..b3317c1 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -49,7 +49,6 @@ const RCA_SCHEMA = { asks_skipped: { type: "array", items: { type: "string" } }, asks_unavailable: { type: "array", items: { type: "string" } }, cluster_id: { type: "string" }, - mandatory_checks: { type: "array", items: { type: "string" } }, }, additionalProperties: true, }; @@ -64,7 +63,6 @@ const shared = [ `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, `Persist eagerly to the CSV: claim your row before turn 1, flip it on terminal (lib/csv-state.mjs).`, - `MANDATORY CONNECTOR SWEEPS (Operating Principle 0, ai-tfa-coordinator.md): before turn 1, check every available capability's connector skill for a declared compulsory check (a connector skill may mark a check "COMPULSORY — not conditional, not a fallback"). Run any that apply NOW, unconditionally, and fold the evidence block into the turn-1 message. Do NOT wait for a NEEDS_INFO ask naming that evidenceType — TFA has been observed to label deploy/infra-shaped questions "product_code", so ask-routing alone will never trigger it. Record what ran under mandatory_checks in the RCA_OUTPUT.`, ].join("\n"); function resumeLine(row) { From 94cb38dda2aaa3763d36175056ccec1864642093 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Thu, 30 Jul 2026 20:44:08 +0530 Subject: [PATCH 05/51] feat(rca): pre-fetch build-level evidence once, share via file across all coordinators Each dispatched ai-tfa-coordinator previously re-ran its own turn-1 PR-window search and kubectl/VictoriaLogs sweep independently, even though that evidence is a property of the build, not of any one test. Add lib/evidence-file.mjs (a build-scoped JSON artifact, keyed by repo and workload, mirroring csv-state.mjs's path convention) so the orchestrator gathers this once and every representative/sibling reads it before falling back to a live call. Validated against a real build: two re-run tests landed the same root causes (including the same culprit PR) at 69-70% fewer tokens and 90-94% fewer tool calls than the original per-coordinator sweep. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 37 ++++++++- lib/evidence-file.mjs | 150 +++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 70 +++++++++++++--- tests/evidence-file.test.mjs | 140 ++++++++++++++++++++++++++++++++ workflows/rca-batch.mjs | 13 ++- 5 files changed, 395 insertions(+), 15 deletions(-) create mode 100644 lib/evidence-file.mjs create mode 100644 tests/evidence-file.test.mjs diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 43f12b7..67e1bf6 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -39,6 +39,12 @@ it names no `kubectl` / `chitragupta` / `bifrost`; it routes by *capability*. - `resume` — optional `{ threadId, turnId }` from a prior PENDING run. - `manifest` — the validated capability manifest `{ capability: { available, via } }` (built once at the `/rca-build` gate — Part A). +- `evidenceFile` — optional. Absolute path to the build-level pre-fetch + artifact (`lib/evidence-file.mjs`, `/rca-build` Step 4). Holds pre-digested + `github` (PR window, deploy state) and `logs`/`infra` (app-side sweep) + evidence, keyed by repo and by workload — gathered ONCE by the orchestrator + for every repo/workload this build's failures implicate. `Read` it before + any live gather call (see Operating Principle 0). If `testRunId` is missing or not parseable as an integer, emit a `failed` `RCA_OUTPUT` block with `root_cause: "no testRunId provided"` and stop — do not @@ -64,6 +70,20 @@ read-only and has no side effects, so a read is always safe to repeat. ## Operating principles +0. **Read the pre-fetch first.** If `evidenceFile` is present, `Read` it + before considering any live github/infra/logs call. It holds build-level + evidence (PR window, deploy state, log sweeps) already gathered once by the + orchestrator for the repos/workloads this build's failures implicate. Use + what it covers directly — its entries are already digest-shaped (an + `evidence-block.md`-style `block`); paste, don't re-digest. Only make a live + call for what it does NOT cover: a repo/workload it doesn't name, an entry + marked with a `gap` (a `gap` is never coverage — treat it exactly as if the + file didn't have that entry), or evidence genuinely specific to this one + test that a build-wide sweep window could plausibly have missed. For a + sibling (`pre_seed` present): the file's data about YOUR OWN test's + workload/repo is real evidence, not inheritance — reading it is fine; the + CONFIRMATION judgment against it must still be independently yours (see + principle 1 and the sibling note in "The loop"). 1. **Logs by TFA — the core contract.** Never seed logs in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or digest log content. Logs are TFA's job. @@ -125,8 +145,10 @@ each candidate suspect **try to disprove it** (path overlap? shipped before the failure window? behind an OFF flag?). Feed both supporting *and* disconfirming evidence back as a structured suspect packet; only `verdict: supported` suspects belong in `related_prs`. Reuse the pre-computed build-level evidence — do not -re-fetch per test. Never fabricate a PR when the github capability is unavailable -— emit an `unavailable` block. +re-fetch per test (the `evidenceFile`'s `github` section, if present and not +`gap`-marked for this repo; otherwise the live github connector). Never +fabricate a PR when the github capability is unavailable — emit an +`unavailable` block. ## The loop @@ -153,8 +175,13 @@ re-fetch per test. Never fabricate a PR when the github capability is unavailabl 3. ROUTE the asks (read references/evidence-routing.md; route via lib/routing.mjs): For each ask, high → medium → low: skip → record in asks_skipped, emit nothing. - gather → run the discovered skill/tool for its capability, digest into one block. - Record evidenceType in asks_fulfilled (dedupe). + gather → FIRST check `evidenceFile` (if present) for this ask's scope — + repo for a github ask, workload for an infra/logs ask. Covered + (present, `gap` falsy) → paste its `block` straight in, no + re-digesting, no live call. Not named in the file, or its + entry has a `gap`, or no `evidenceFile` at all → run the + discovered skill/tool live, exactly as before. + Digest into one block. Record evidenceType in asks_fulfilled (dedupe). gap → emit an `unavailable` block (record in asks_unavailable). NEVER prompt. PRODUCT_BUG in play + no supported PR yet → widen the github hunt this turn. Concatenate per-ask blocks into the next-turn MESSAGE (respect size caps). @@ -244,6 +271,8 @@ Notes: ## Hard limits +- **Never** treat a `gap`-marked `evidenceFile` entry as coverage — a `gap` + means attempt a live call exactly as if the file didn't have that entry. - **Never** prompt, ask, or wait on a user — the gate is closed; gaps degrade to `unavailable`. - **Never** fulfill or seed a `test_logs` ask — TFA owns logs. - **Never** exceed `turnCap` `tfaRcaTurn` calls in one run. diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs new file mode 100644 index 0000000..f6eb55c --- /dev/null +++ b/lib/evidence-file.mjs @@ -0,0 +1,150 @@ +// Build-level evidence pre-fetch artifact (see docs/plan: evidence-file). PR +// windows, deploy state, and app-log sweeps are properties of the BUILD, not +// of any one test — a naive batch re-fetches them once per dispatched +// coordinator. This module persists them to a file ONCE so every +// representative and sibling `ai-tfa-coordinator` dispatch can `Read` the same +// artifact instead of re-running the same `gh`/`kubectl`/`grafana` calls. +// +// Layered under `lib/evidence-cache.mjs`, not merged with it: the cache is an +// in-process, function-scoped Map that dedups compute *within* the +// orchestrator's own Step 4 pass; this module is what makes that result +// visible to OTHER processes (the independently-dispatched coordinator +// subagents, which share no memory with the orchestrator or each other). +// +// Path convention mirrors `lib/csv-state.mjs`'s `csvPathFor` exactly: the +// build id is in the filename (no cross-build collisions) and the default +// directory is OS temp (`<tmpdir>/bstack-rca/`), so a build's evidence file +// sits right next to its state CSV. `stateDir` overrides the directory only. +// +// Invariant: this file NEVER carries `test_logs` content. `logs` is keyed by +// *workload* (an infra/pod concept), populated only via the `infra`/`logs` +// capability — TFA remains the sole owner of test-side SDK/driver/session +// logs, which structurally cannot land here. +// +// Timestamps are passed in as `nowMs` (never read from the clock here), same +// discipline as `csv-state.mjs`, so this stays usable from the Workflow-tool +// sandbox (which forbids `Date.now()`). + +import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { tmpdir } from "node:os"; + +/** Canonical evidence-file path for one build's run. Same two invariants as + * `csvPathFor`: build id in the filename; OS temp by default; `stateDir` + * overrides the directory only. */ +export function evidencePathFor(buildId, stateDir = "") { + const safe = String(buildId ?? "").replace(/[^A-Za-z0-9._-]/g, "_") || "unknown-build"; + const dir = stateDir && String(stateDir).trim() !== "" ? String(stateDir) : join(tmpdir(), "bstack-rca"); + return join(dir, `rca-evidence.${safe}.json`); +} + +export function emptyEvidenceFile(buildId, nowMs) { + return { + buildId: String(buildId ?? ""), + generatedAtMs: nowMs, + baseline: null, + suspectWindow: null, + github: {}, + logs: {}, + coverage: { reposCovered: [], reposGapped: [], workloadsCovered: [], workloadsGapped: [] }, + }; +} + +/** Read-only; never throws on a missing file — a coordinator (or the + * orchestrator, before Step 4 has run) always gets a well-shaped, empty-covered + * result rather than an exception. Graceful degradation is the point: an + * absent/partial file just means every ask falls back to a live gather. */ +export function readEvidenceFile(filePath) { + if (!existsSync(filePath)) return emptyEvidenceFile("unknown-build", 0); + return JSON.parse(readFileSync(filePath, "utf8")); +} + +export function writeEvidenceFile(filePath, doc) { + const dir = dirname(filePath); + if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true }); + writeFileSync(filePath, JSON.stringify(doc, null, 2), "utf8"); +} + +function loadOrInit(filePath, nowMs) { + if (!existsSync(filePath)) return emptyEvidenceFile("unknown-build", nowMs); + return readEvidenceFile(filePath); +} + +/** Idempotent: creates the file with the given `buildId` if it doesn't exist + * yet, otherwise leaves an existing file untouched (never clobbers prior + * writes on a resume). Call this FIRST, before any `set*` call, so `buildId` + * is recorded correctly — the `set*` functions below fall back to + * `"unknown-build"` only as a safety net if called without this. */ +export function initEvidenceFile(filePath, buildId, nowMs) { + if (existsSync(filePath)) return readEvidenceFile(filePath); + const doc = emptyEvidenceFile(buildId, nowMs); + writeEvidenceFile(filePath, doc); + return doc; +} + +/** Records the diff/PR-window baseline once, at the start of the Step 4 pass. + * `baseline` is `resolveBaseline(...)`'s return value from `evidence-cache.mjs` + * (`{ref, isFallback}`); `suspectWindow` is whatever shape the active connector + * skill uses to describe the window (e.g. `{reposRequested, startedAt}`). */ +export function setBaseline(filePath, baseline, suspectWindow, nowMs) { + const doc = loadOrInit(filePath, nowMs); + doc.baseline = baseline; + doc.suspectWindow = suspectWindow; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return doc; +} + +/** Read-modify-write merge into `doc.github[repo]`. `entry` shape: + * `{ deployState: {block, gap}, prsInWindow: [{pr, files, block, verdict}], + * gap }` — `gap` (top-level, on the repo entry) is what `recomputeCoverage` + * checks; a repo present with a non-null `gap` is NOT counted as covered. + * Only ever touches this one repo's key — every other repo/workload already + * in the file is untouched. */ +export function setGithubEvidence(filePath, repo, entry, nowMs) { + const doc = loadOrInit(filePath, nowMs); + doc.github[repo] = entry; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return doc; +} + +/** Read-modify-write merge into `doc.logs[workload]`. `entry` shape: + * `{ clusterIds, kubectlSweep: {block, gap}, victorialogs: {block, gap}, gap }`. + * Same no-clobber guarantee as `setGithubEvidence`, keyed by workload instead + * of repo. */ +export function setLogsEvidence(filePath, workload, entry, nowMs) { + const doc = loadOrInit(filePath, nowMs); + doc.logs[workload] = entry; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return doc; +} + +// A requested item is "covered" only if it is present AND its own `gap` field +// is falsy. Presence with a `gap` is a recorded, deliberate miss — not +// coverage — so a coordinator (or this function) never mistakes "we looked +// and couldn't get it" for "we have it." +function isCovered(doc, section, key) { + const entry = doc[section]?.[key]; + return Boolean(entry) && !entry.gap; +} + +/** Derives `doc.coverage` from exactly which requested repos/workloads have a + * gap-free entry, and persists it. `requested = {repos:[...], workloads:[...]}` + * — normally the Gate Part A scope-probe-validated repo list and the union of + * workloads every cluster's representative implicates (see SKILL.md Step 4). */ +export function recomputeCoverage(filePath, requested, nowMs) { + const doc = loadOrInit(filePath, nowMs); + const repos = requested?.repos ?? []; + const workloads = requested?.workloads ?? []; + doc.coverage = { + reposCovered: repos.filter((r) => isCovered(doc, "github", r)), + reposGapped: repos.filter((r) => !isCovered(doc, "github", r)), + workloadsCovered: workloads.filter((w) => isCovered(doc, "logs", w)), + workloadsGapped: workloads.filter((w) => !isCovered(doc, "logs", w)), + }; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return doc.coverage; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 581e834..9fc9dca 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -226,15 +226,54 @@ Each cluster gets one **representative** (full multi-turn loop) and `N−1` the expensive evidence hunt to O(distinct causes) while every test still lands a per-test RCA. Singleton clusters are just plain per-test loops. -## Step 4 — build-evidence pre-compute (see references/evidence-routing.md) - -Once, before fan-out (the capability manifest already exists from Gate Part A — -reuse it, do not re-discover): - -- **Build-level evidence** — compute the last-green→this-build delta (diff, - deploy timeline, suspect-PR window) **once** and pre-seed every coordinator - with the same grounded window. Cache by `(repo, commit-range)`. No "last green" - baseline (never-green suite) → fall back to a configured baseline ref and log it. +## Step 4 — build-evidence pre-fetch (see references/evidence-routing.md and lib/evidence-file.mjs) + +Once, after clustering (Step 3) and before fan-out — the capability manifest +already exists from Gate Part A, reuse it, do not re-discover. This step +replaces each coordinator's own turn-1 evidence sweep with ONE pre-fetch: +it does not remove the requirement that turn-1 evidence exists, only *who +gathers it*. + +1. Resolve the evidence-file path: `lib/evidence-file.mjs` → + `evidencePathFor(buildId, config.paths.stateDir)` — + `<tmpdir>/bstack-rca/rca-evidence.<buildId>.json`, alongside the state CSV. + `initEvidenceFile(path, buildId, nowMs)`. +2. **Scope the pre-fetch to the full union, never a single guess:** + - **Repos** — every repo in Gate Part A's scope-probe-validated + `repos_validated` list (e.g. a VRT-lane build validates `frontend` + + `railsApp`; an nl2steps build validates `misc-services` + `ai-sdk-node`). + - **Workloads** — the union of workloads every cluster's **representative** + implicates, via the active connector skill's failure-signature→workload + routing table (never one workload guessed from the first failing test). +3. For each repo: run the connector skill's PR-window-search + deploy-state + recipes **once**, using `lib/evidence-cache.mjs`'s `compute(repo, range, + evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. + Digest the result into the `evidence-block.md` shape, then persist via + `setGithubEvidence(path, repo, {deployState, prsInWindow, gap}, nowMs)`. + A repo the connector can't reach records `{gap: "<reason>"}` — never blocks + the rest of the pre-fetch. +4. For each workload: run the connector skill's compulsory kubectl + + VictoriaLogs sweep **once**, scoped to the build's own failure window + (`started_at`..`finished_at`, not "now" — see the connector skill's window + guidance). Persist via `setLogsEvidence(path, workload, + {clusterIds, kubectlSweep, victorialogs, gap}, nowMs)`. +5. `resolveBaseline(lastGreenRef, fallbackRef)` (from `lib/evidence-cache.mjs`) + → `setBaseline(path, baseline, suspectWindow, nowMs)`. No "last green" + baseline (never-green suite) → fall back to a configured baseline ref and + note the weaker grounding — this note travels into the file, not just a + spoken log line, so every coordinator sees it. +6. `recomputeCoverage(path, {repos, workloads}, nowMs)` and declare the + resulting path in the gate summary alongside the capability manifest, so + a human re-reading the run can find it. + +**Size discipline is enforced at write time, not just at submit time.** Every +leaf (`deployState`, each PR, each log sweep) must already be a digested +`block` per `evidence-routing.md`'s caps (`SUMMARY≤400`, `SNIPPET≤20/40 lines`, +link over diff) — never a raw dump. Cap `prsInWindow` to the top ~30 candidates +by path-overlap relevance, not every PR in the window. + +Pass `evidencePathFor(...)`'s path to Step 5's fan-out as `evidenceFilePath` — +every dispatch (representative and sibling) must be told to read it first. ## Step 5 — fan-out (fully autonomous) @@ -284,6 +323,19 @@ therefore lets the coordinator infer repos from workspace `git remote` or cwd — is a bug: the coordinator will land plausible-but-wrong PR attributions on adjacent repos. +**Coordinator prompts MUST also name the Step 4 evidence file.** Every +dispatch prompt (representative and sibling alike) includes the absolute +`evidenceFilePath` from Step 4 with the instruction: *"Read `<path>` (via the +Read tool) before making any live github/infra/logs gather call. It's a +pre-fetch, not a hard dependency — a repo/workload it doesn't name, or marks +with a `gap`, is a genuine gap: fall back to the capability manifest above +exactly as if no file existed."* For a sibling, add: *"The file's data about +your OWN test's workload is real evidence, not inheritance — reading it is +fine. What must stay independent is the CONFIRMATION judgment: never adopt the +representative's verdict just because the file already has the answer in +it."* A dispatch prompt that omits this path forces its coordinator back into +a full independent sweep — exactly the redundancy Step 4 exists to remove. + **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs new file mode 100644 index 0000000..a01466a --- /dev/null +++ b/tests/evidence-file.test.mjs @@ -0,0 +1,140 @@ +import { test, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + evidencePathFor, + emptyEvidenceFile, + initEvidenceFile, + readEvidenceFile, + writeEvidenceFile, + setBaseline, + setGithubEvidence, + setLogsEvidence, + recomputeCoverage, +} from "../lib/evidence-file.mjs"; + +let dir; +let file; + +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "rca-evidence-")); + file = join(dir, "evidence.json"); +}); +afterEach(() => rmSync(dir, { recursive: true, force: true })); + +test("evidencePathFor: build id is in the filename, default dir is OS temp", () => { + const p = evidencePathFor("abc123XYZ"); + assert.ok(p.startsWith(join(tmpdir(), "bstack-rca"))); + assert.ok(p.endsWith("rca-evidence.abc123XYZ.json")); +}); + +test("evidencePathFor: different builds never share a path", () => { + assert.notEqual(evidencePathFor("build-A"), evidencePathFor("build-B")); +}); + +test("evidencePathFor: sanitizes hostile ids and handles empty", () => { + assert.ok(evidencePathFor("../../etc/passwd").endsWith("rca-evidence..._.._etc_passwd.json")); + assert.ok(evidencePathFor("").endsWith("rca-evidence.unknown-build.json")); +}); + +test("evidencePathFor: stateDir override wins over temp", () => { + const p = evidencePathFor("b1", "/ci/artifacts"); + assert.equal(p, join("/ci/artifacts", "rca-evidence.b1.json")); +}); + +test("readEvidenceFile on a missing path returns the empty shape, never throws", () => { + const doc = readEvidenceFile(file); + assert.deepEqual(doc, emptyEvidenceFile("unknown-build", 0)); +}); + +test("initEvidenceFile creates the file with the given buildId", () => { + const doc = initEvidenceFile(file, "build-1", 1000); + assert.equal(doc.buildId, "build-1"); + assert.equal(doc.generatedAtMs, 1000); + assert.deepEqual(readEvidenceFile(file), doc); +}); + +test("initEvidenceFile is idempotent — does not clobber an existing file", () => { + initEvidenceFile(file, "build-1", 1000); + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "x" } }, 2000); + const before = readEvidenceFile(file); + const again = initEvidenceFile(file, "build-1", 9999); + assert.deepEqual(again, before); +}); + +test("setGithubEvidence and setLogsEvidence coexist without clobbering each other", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "a-deploy" } }, 1000); + setLogsEvidence(file, "workload-1", { gap: null, kubectlSweep: { block: "w1-logs" } }, 1000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].deployState.block, "a-deploy"); + assert.equal(doc.logs["workload-1"].kubectlSweep.block, "w1-logs"); +}); + +test("setGithubEvidence for a second repo does not disturb the first", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "a" } }, 1000); + setGithubEvidence(file, "org/b", { gap: null, deployState: { block: "b" } }, 1000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].deployState.block, "a"); + assert.equal(doc.github["org/b"].deployState.block, "b"); +}); + +test("setGithubEvidence twice for the SAME repo overwrites only that repo", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "old" } }, 1000); + setGithubEvidence(file, "org/b", { gap: null, deployState: { block: "b" } }, 1000); + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "new" } }, 2000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].deployState.block, "new"); + assert.equal(doc.github["org/b"].deployState.block, "b"); // untouched +}); + +test("setBaseline records baseline and suspectWindow without touching github/logs", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "a" } }, 1000); + setBaseline(file, { ref: "sha123", isFallback: false }, { reposRequested: ["org/a"] }, 2000); + const doc = readEvidenceFile(file); + assert.deepEqual(doc.baseline, { ref: "sha123", isFallback: false }); + assert.deepEqual(doc.suspectWindow, { reposRequested: ["org/a"] }); + assert.equal(doc.github["org/a"].deployState.block, "a"); // untouched +}); + +test("recomputeCoverage: a covered repo/workload has no gap; a missing one is gapped", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "a" } }, 1000); + setLogsEvidence(file, "w1", { gap: null, kubectlSweep: { block: "w1" } }, 1000); + const coverage = recomputeCoverage( + file, + { repos: ["org/a", "org/b"], workloads: ["w1", "w2"] }, + 2000, + ); + assert.deepEqual(coverage.reposCovered, ["org/a"]); + assert.deepEqual(coverage.reposGapped, ["org/b"]); + assert.deepEqual(coverage.workloadsCovered, ["w1"]); + assert.deepEqual(coverage.workloadsGapped, ["w2"]); +}); + +test("recomputeCoverage: a present entry with a non-null gap is NOT covered", () => { + setGithubEvidence(file, "org/a", { gap: "gh auth failed for this repo" }, 1000); + const coverage = recomputeCoverage(file, { repos: ["org/a"], workloads: [] }, 2000); + assert.deepEqual(coverage.reposCovered, []); + assert.deepEqual(coverage.reposGapped, ["org/a"]); +}); + +test("recomputeCoverage persists onto the file (readable afterwards)", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "a" } }, 1000); + recomputeCoverage(file, { repos: ["org/a"], workloads: [] }, 2000); + const doc = readEvidenceFile(file); + assert.deepEqual(doc.coverage.reposCovered, ["org/a"]); +}); + +test("a block string with newlines and quotes round-trips through JSON unchanged", () => { + const block = 'ASK: did X change?\nTYPE: product_code\nFOUND: yes\nSUMMARY: "quoted" finding\nSNIPPET: line1\nline2'; + setGithubEvidence(file, "org/a", { gap: null, deployState: { block } }, 1000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].deployState.block, block); +}); + +test("writeEvidenceFile creates the parent directory if missing", () => { + const nested = join(dir, "nested", "sub", "evidence.json"); + writeEvidenceFile(nested, emptyEvidenceFile("build-1", 0)); + assert.deepEqual(readEvidenceFile(nested).buildId, "build-1"); +}); diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index b3317c1..1624b54 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -23,7 +23,13 @@ export const meta = { // { // csvPath, buildId, // manifest: { capability: { available, via } }, -// buildEvidence: { baselineRef, suspectWindow, ... }, // pre-computed once +// evidenceFilePath, // NEW — lib/evidence-file.mjs artifact for this build +// buildEvidence: { baselineRef, isFallback, suspectWindow, reposCovered, workloadsCovered, gaps }, +// // ^ SHRUNK to a cheap summary/pointer only — the full PR list / log +// // sweeps live in the file at evidenceFilePath, read via each +// // coordinator's own Read tool. Repeating the full detail in every +// // dispatch prompt (as before) is exactly the duplication this file +// // removes. // clusters: [ // { cluster_id, representative: { testRunId, testName, error_summary }, // siblings: [ { testRunId, testName, error_summary } ] } @@ -58,7 +64,9 @@ const clusters = ctx.clusters ?? []; const shared = [ `CSV state file: ${ctx.csvPath}`, `Capability manifest: ${JSON.stringify(ctx.manifest ?? {})}`, - `Build-level evidence (pre-computed once, reuse — do not re-fetch): ${JSON.stringify(ctx.buildEvidence ?? {})}`, + `Pre-fetched build-evidence file — READ THIS FIRST (via the Read tool) before making ANY live github/infra/logs gather call: ${ctx.evidenceFilePath}`, + `Build-evidence summary (full detail is in the file above; this is only a pointer — do not re-fetch what the file already covers): ${JSON.stringify(ctx.buildEvidence ?? {})}`, + `If the file's github/logs sections do not name a repo/workload/ask you need, or record a "gap" for it, that is a genuine gap — fall back to a live gather via the capability manifest above exactly as if no file existed. The file is an optimization, never a hard dependency.`, `Autonomous run — on an evidence gap with no valid connector, report "unavailable" back to TFA (NEVER prompt a user). Best-effort finalize.`, `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, @@ -98,6 +106,7 @@ function siblingPrompt(sibling, repResult, cluster) { ` root_cause: ${repResult?.root_cause ?? "(representative did not resolve)"}`, ` related_prs: ${JSON.stringify(repResult?.related_prs ?? [])}`, `State this hypothesis on turn 1 and ask TFA to CONFIRM it against THIS test's own logs.`, + `The pre-fetched evidence file's data about your OWN workload is real evidence about YOUR OWN test — reading it is NOT blind inheritance. What must stay independent is the CONFIRMATION judgment: never adopt the representative's verdict just because the file already has the answer in it.`, `If TFA confirms in one turn → done. If it does NOT (NEEDS_INFO), fall back to the full loop — never blindly inherit.`, `testRunId=${sibling.testRunId} testName=${sibling.testName ?? ""}`, `error_digest: ${sibling.error_summary ?? "(none)"}`, From 4a3912acf93a05d09935faf6c0c6cb7900b01843 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Fri, 31 Jul 2026 10:59:46 +0530 Subject: [PATCH 06/51] updating prompts --- agents/ai-tfa-coordinator.md | 16 ++++++++++ .../rca-build/references/github-evidence.md | 29 +++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 67e1bf6..64abace 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -117,6 +117,18 @@ read-only and has no side effects, so a read is always safe to repeat. 8. **Never editorialize.** Report findings (suspect PR, server-side error line), not verdicts. The root cause is TFA's to state on `RESOLVED`; pass its `glimpse` through verbatim. +9. **Field-filter every gather call, always.** Before running any + capability-provided command (`gh`, `kubectl`, or whatever the manifest + resolved to for `github`/`infra`), project down to only the field(s) this + ask needs — `--jq`, `-o custom-columns`, `-o jsonpath`, or a `grep`/`head` + immediately piped. Never run the unfiltered form "just to see the shape" — + an exploratory call costs the same context whether or not its output ends + up in the digest, and a raw repo/commit/pod object typically carries + orders of magnitude more noise (license/URL metadata, multi-hundred-char + signature blocks, unrequested columns) than any evidence ask ever uses. + This governs what enters *your own* context via the tool result — distinct + from principle 6, which governs the digest you send back to TFA. Exact + command templates: `references/github-evidence.md` § Field-filtering. ## Application bugs — the culprit-PR mandate (MANDATORY) @@ -282,6 +294,10 @@ Notes: - **Never** let drain reads consume the turn cap, and never drain past the `softPendingDrain` budget — a wedged turn must not hang the batch. - **Never** dump raw logs, full diffs, or full file contents into a turn message — digest only. +- **Never** run an unfiltered gather call (a bare `gh api ...` with no `--jq`, + `kubectl get ... -o wide`/`-o yaml` when a narrower `-o custom-columns` + answers the ask) — project to the needed field(s) before the call runs, not + by reading past the noise after. - **Never** write to any repo / cluster / ticket / the run — every action is read-only. - **Never** editorialize a cause — pass TFA's `glimpse` through verbatim. - **Never** blindly inherit a representative's cause for a sibling — confirm against its own logs. diff --git a/skills/rca-build/references/github-evidence.md b/skills/rca-build/references/github-evidence.md index 4b2c445..a3d6b3f 100644 --- a/skills/rca-build/references/github-evidence.md +++ b/skills/rca-build/references/github-evidence.md @@ -55,6 +55,35 @@ Scope everything by the failing test's `file_path` + the error summary. The build-level evidence (diff-since-last-green, PR window) is **pre-computed once** and passed in — reuse it; do not re-fetch per test. +## Field-filtering — project before you pull, every call + +The single most common way a gather call wastes context: pulling a full +object when the ask only needs one or two fields from it. This applies to +whichever connector resolved for `github` (most commonly the `gh` CLI today, +or a GitHub MCP tool) — every call should already be filtered to the field(s) +the ask needs, not filtered after the fact by reading past the noise. The +same discipline applies to `infra` gather calls (`kubectl` or whatever the +manifest resolved to), since the failure mode is identical. + +| Need | Don't — pulls the whole object | Do — projects to the field(s) the ask needs | +|---|---|---| +| Repo exists / default branch | `gh api repos/OWNER/REPO` | `gh api repos/OWNER/REPO --jq '.default_branch'` | +| Branch exists on the shipping branch | `gh api repos/OWNER/REPO/branches/BRANCH` | `gh api repos/OWNER/REPO/branches/BRANCH --jq '.name'` | +| Commit history / PR-window search | `gh api "repos/OWNER/REPO/commits?sha=BRANCH&per_page=100"` | add `--jq '[.[] | {sha: .sha[0:8], date: .commit.committer.date, msg: (.commit.message | split("\n")[0])}]'` | +| PR metadata | `gh pr view N --repo OWNER/REPO` (full payload) | `gh pr view N --repo OWNER/REPO --json state,mergedAt,baseRefName,headRefOid,files,author` — `--json` is itself a field allowlist; list only the fields this ask uses | +| Pod / workload listing | `kubectl get pods -n NS -o wide` | `kubectl get pods -n NS -o custom-columns='NAME:.metadata.name,STATUS:.status.phase'` | +| Deploy / image state | `kubectl get deploy -n NS -o yaml` | `kubectl get deploy -n NS -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'` | +| Log sweep | a raw `--tail` dump | `kubectl logs POD --since=<window> --tail=2000 \| grep -E '<correlation token>\|ERROR\|Exception'` — filter by the correlation token, never a raw tail | + +**Never run the unfiltered form "to see the shape first."** An exploratory +raw call costs the same context whether or not its output ends up in the +digest — a bare repo or commit object routinely carries license/URL metadata +and a multi-hundred-character signature block that no evidence ask ever +consults. If the exact field path is genuinely unknown, learn the shape from +one throwaway call against a cheap target, then filter every real call from +that point on — never repeat the unfiltered form per repo, per PR, or per +test. + ## Falsification protocol — rule out, don't just rule in For **each** candidate suspect PR, try to **break** the hypothesis: From 1eba07ff8b420a75fe12cc91a33e3817f82643f7 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 12:56:44 +0530 Subject: [PATCH 07/51] fix(rca): tighten turnMessageMaxChars to 1000 Self-imposed plugin budget, well under the tfaRcaTurn tool's actual 5000-char hard cap. Retunes the per-block digest caps (SUMMARY, SNIPPET) proportionally so real turn-1 messages fit the new budget instead of relying on ad-hoc truncation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 4 +++- config/rca.config.json | 2 +- skills/rca-build/SKILL.md | 2 +- skills/rca-build/references/evidence-routing.md | 17 +++++++++++------ skills/rca-build/templates/evidence-block.md | 6 +++--- 5 files changed, 19 insertions(+), 12 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 64abace..19c907a 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -110,7 +110,9 @@ read-only and has no side effects, so a read is always safe to repeat. 6. **Digest, don't dump.** Every follow-up `message` carries digested findings (`ask → found → snippet/link`), never raw log tails, full diffs, or full files. Size caps + block shape live in `references/evidence-routing.md` — read it - before fulfilling any ask. The tool caps `message` at 5000 chars. + before fulfilling any ask. The plugin config caps `message` at 1000 chars + (`turnMessageMaxChars` in `config/rca.config.json`); the `tfaRcaTurn` tool + itself would allow up to 5000, but the plugin self-limits to 1000. 7. **Report gaps, don't drop them.** An ask the coordinator cannot fulfill becomes a `not-found` / `unreachable` / `unavailable` block, never a silent omission — and **never a user prompt**. TFA finalizes best-effort with lower confidence. diff --git a/config/rca.config.json b/config/rca.config.json index 45c3997..b316f94 100644 --- a/config/rca.config.json +++ b/config/rca.config.json @@ -4,7 +4,7 @@ "$concurrencyComment": "Max coordinator subagents run in parallel during Step 5 fan-out. HONORED LITERALLY on the default path (direct Agent-tool dispatch of ai-tfa-coordinator subagents — one message, up to `concurrency` tool-use blocks per batch) and by the sequential harness (lib/loop.mjs). Only a SOFT target on the opt-in Workflow-tool path (workflows/rca-batch.mjs), which the Workflow runtime hard-caps at min(16, cpu cores - 2) regardless of this value — that cap is architectural and cannot be raised from this repo. See SKILL.md Step 5.", "concurrency": 20, "turnCap": 6, - "turnMessageMaxChars": 5000, + "turnMessageMaxChars": 1000, "pollSoftPendingMs": 90000, "$softPendingDrainComment": "tfaRcaTurn abandons its in-call poll at pollSoftPendingMs (90s) and returns a soft PENDING while the TFA agent keeps working — turns finalizing past 90s are routine. On a soft PENDING the loop READS the same turnId via getTfaTurnResult on this budget before it routes asks or submits anything further; reads do not consume turnCap. Only when the budget is spent does the run end PENDING (pending-resume row).", "softPendingDrain": { diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 9fc9dca..0f9f348 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -268,7 +268,7 @@ gathers it*. **Size discipline is enforced at write time, not just at submit time.** Every leaf (`deployState`, each PR, each log sweep) must already be a digested -`block` per `evidence-routing.md`'s caps (`SUMMARY≤400`, `SNIPPET≤20/40 lines`, +`block` per `evidence-routing.md`'s caps (`SUMMARY≤80`, `SNIPPET≤4/8 lines`, link over diff) — never a raw dump. Cap `prsInWindow` to the top ~30 candidates by path-overlap relevance, not every PR in the window. diff --git a/skills/rca-build/references/evidence-routing.md b/skills/rca-build/references/evidence-routing.md index 05b4435..9132fef 100644 --- a/skills/rca-build/references/evidence-routing.md +++ b/skills/rca-build/references/evidence-routing.md @@ -89,16 +89,21 @@ and unfulfillable variants) — copy it, don't retype it. Shape: | Field / scope | Soft target | Hard ceiling | On exceed | |---|---|---|---| -| `SUMMARY` | ≤ 300 chars | 400 chars | Tighten to the finding; drop restatement of the ask | -| `SNIPPET` per ask | ≤ 20 lines | 40 lines | Keep the load-bearing lines; replace the rest with `… (N lines elided — see LINK)` | -| Code diff in a `product_code` snippet | ≤ 1 hunk | 3 hunks | Show changed lines + 3 lines context; link the full PR | -| Whole next-turn `message` | ≤ 200 lines | 400 lines (and ≤ `turnMessageMaxChars`) | Drop `low`-priority asks first; keep every `high` ask's block | +| `SUMMARY` | ≤ 60 chars | 80 chars | Tighten to the finding; drop restatement of the ask | +| `SNIPPET` per ask | ≤ 4 lines | 8 lines | Keep the load-bearing lines; replace the rest with `… (N lines elided — see LINK)` | +| Code diff in a `product_code` snippet | ≤ 1 hunk | 2 hunks | Show changed lines only, no context lines; link the full PR | +| Whole next-turn `message` | ≤ 40 lines | 80 lines (and ≤ `turnMessageMaxChars`) | Drop `low`-priority asks first; keep every `high` ask's block | | Asks fulfilled per turn | all `high` + `medium` | — | Defer `low` asks to a later turn rather than truncating a `high` ask | Truncation rule of thumb: **never truncate a `high`-priority ask's block to fit a `low`-priority one.** Drop the low block whole; keep the high block intact. The -whole-message ceiling also honors `turnMessageMaxChars` from -`config/rca.config.json` (the tool caps `message` at 5000 chars). +whole-message ceiling honors `turnMessageMaxChars` from +`config/rca.config.json`, now set to **1000 chars** — a plugin-configured +self-limit, tighter than the underlying tool's actual hard cap (the +`tfaRcaTurn` MCP tool itself allows up to 5000 chars per `message`; the plugin +just chooses not to use all of it). At this budget, expect at most 2-3 ask +blocks per turn before hitting the ceiling — defer lower-priority asks to a +follow-up turn rather than cramming everything into one. ### What never goes in a digest diff --git a/skills/rca-build/templates/evidence-block.md b/skills/rca-build/templates/evidence-block.md index fa0bf49..97b41f7 100644 --- a/skills/rca-build/templates/evidence-block.md +++ b/skills/rca-build/templates/evidence-block.md @@ -6,12 +6,12 @@ the forbidden list: `../references/evidence-routing.md`. Fulfilled ask: ``` -ASK: <verbatim `what` from the TfaAsk, ≤ 120 chars> +ASK: <verbatim `what` from the TfaAsk, ≤ 80 chars> TYPE: <evidenceType> FOUND: <yes | no | partial> -SUMMARY: <1–3 sentences — the finding, in the agent's words. ≤ 400 chars> +SUMMARY: <1 sentence — the finding, in the agent's words. ≤ 80 chars> SNIPPET: - <the load-bearing excerpt only — see size caps. Omit if a LINK fully carries it.> + <the load-bearing excerpt only, ≤ 4 lines — see size caps. Omit if a LINK fully carries it.> LINK: <permalink to the source — PR/commit/log-search/metrics panel/deploy record. Omit if N/A.> ``` From 70ef859f28ef10f6406afd44a2178a927a691d28 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 16:08:59 +0530 Subject: [PATCH 08/51] feat(rca): let coordinators write gathered evidence back to the shared file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pre-fetch file was read-only from a coordinator's side — when a live gather filled a gap or went deeper than the pre-fetch had (a full diff instead of a summary, a PR the pre-fetch never named), that extra work died with the coordinator instead of benefiting its own siblings or other clusters sharing the same repo/workload. Add mergeGithubEvidence/mergeLogsEvidence (read-modify-write, dedupe PRs by number, union clusterIds, never drops what a patch doesn't mention) and wire Operating Principle 0 + the culprit-PR mandate + the routing step in ai-tfa-coordinator.md to call them after any live gather. Confirmed on a real prior run (vrt_345): a representative's own deep PR-diff investigation was exactly the kind of work its sibling had no way to reuse under the read-only version of this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 34 +++++++++++++++-- lib/evidence-file.mjs | 70 +++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 12 ++++++ tests/evidence-file.test.mjs | 71 ++++++++++++++++++++++++++++++++++++ workflows/rca-batch.mjs | 1 + 5 files changed, 184 insertions(+), 4 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 19c907a..9a43df1 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -44,7 +44,11 @@ it names no `kubectl` / `chitragupta` / `bifrost`; it routes by *capability*. `github` (PR window, deploy state) and `logs`/`infra` (app-side sweep) evidence, keyed by repo and by workload — gathered ONCE by the orchestrator for every repo/workload this build's failures implicate. `Read` it before - any live gather call (see Operating Principle 0). + any live gather call (see Operating Principle 0) — and treat it as + read-WRITE: a live gather that fills a gap or goes deeper is written back + via `mergeGithubEvidence`/`mergeLogsEvidence` so later dispatches (this + test's own siblings, or another cluster sharing the same repo/workload) + benefit too. If `testRunId` is missing or not parseable as an integer, emit a `failed` `RCA_OUTPUT` block with `root_cause: "no testRunId provided"` and stop — do not @@ -84,6 +88,21 @@ read-only and has no side effects, so a read is always safe to repeat. workload/repo is real evidence, not inheritance — reading it is fine; the CONFIRMATION judgment against it must still be independently yours (see principle 1 and the sibling note in "The loop"). + + **Write back what you gather live.** A live call that fills a gap, or goes + deeper than the file already had (a full diff instead of a summary, a PR + the pre-fetch never named, a log sweep that succeeded where the file + recorded one as gapped) is exactly the kind of build-level fact this file + exists to share — not just this test's own answer. Persist it via + `mergeGithubEvidence(evidenceFilePath, repo, patch, nowMs)` or + `mergeLogsEvidence(evidenceFilePath, workload, patch, nowMs)` + (`lib/evidence-file.mjs`) before finishing this test, so a sibling + dispatched after you (or any other cluster that turns out to share the + same repo/workload) reads the enriched entry instead of re-fetching what + you just fetched. Only write back genuinely new/deeper findings — don't + write back a no-op read of an already-covered entry. This has the same + informal-locking caveat as the CSV: it's a best-effort optimization, not a + correctness dependency, so never block or retry on it. 1. **Logs by TFA — the core contract.** Never seed logs in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or digest log content. Logs are TFA's job. @@ -160,8 +179,12 @@ failure window? behind an OFF flag?). Feed both supporting *and* disconfirming evidence back as a structured suspect packet; only `verdict: supported` suspects belong in `related_prs`. Reuse the pre-computed build-level evidence — do not re-fetch per test (the `evidenceFile`'s `github` section, if present and not -`gap`-marked for this repo; otherwise the live github connector). Never -fabricate a PR when the github capability is unavailable — emit an +`gap`-marked for this repo; otherwise the live github connector). A culprit +hunt often needs to go deeper than the file's summary — a full diff, a +downstream consumer of a changed flag — write that depth back via +`mergeGithubEvidence` once found, so a sibling confirming the same suspect PR +doesn't re-run the same diff/search. Never fabricate a PR when the github +capability is unavailable — emit an `unavailable` block. ## The loop @@ -194,7 +217,10 @@ fabricate a PR when the github capability is unavailable — emit an (present, `gap` falsy) → paste its `block` straight in, no re-digesting, no live call. Not named in the file, or its entry has a `gap`, or no `evidenceFile` at all → run the - discovered skill/tool live, exactly as before. + discovered skill/tool live, exactly as before — THEN write the + result back via `mergeGithubEvidence`/`mergeLogsEvidence` + (Operating Principle 0) so this fills the gap for whoever + reads the file next. Digest into one block. Record evidenceType in asks_fulfilled (dedupe). gap → emit an `unavailable` block (record in asks_unavailable). NEVER prompt. PRODUCT_BUG in play + no supported PR yet → widen the github hunt this turn. diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index f6eb55c..d7487e5 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -24,6 +24,21 @@ // Timestamps are passed in as `nowMs` (never read from the clock here), same // discipline as `csv-state.mjs`, so this stays usable from the Workflow-tool // sandbox (which forbids `Date.now()`). +// +// Write-back (mergeGithubEvidence / mergeLogsEvidence): the orchestrator's +// Step 4 pass is not the only writer. A coordinator that had to gather live +// (a repo/PR/workload the pre-fetch didn't cover, or covered only with a +// summary) should write what it found back into this SAME file — a +// representative's deep dive then benefits its own siblings (dispatched +// after it resolves) and any other cluster that turns out to share the same +// repo/workload, without re-fetching. Concurrency caveat, same philosophy as +// `csv-state.mjs`'s "true multi-process locking is out of scope": this is a +// synchronous read-modify-write with no file lock, so two coordinators +// writing to the SAME repo/workload key at truly the same moment can lose an +// update. In practice this is low-risk for the case this exists to serve — +// a sibling is dispatched only after its representative resolves, i.e. +// sequentially, never concurrently with it — and acceptable for the rarer +// case of two parallel representatives happening to touch the same repo. import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs"; import { dirname, join } from "node:path"; @@ -121,6 +136,61 @@ export function setLogsEvidence(filePath, workload, entry, nowMs) { return doc; } +// Read-modify-write, preserving whatever isn't in `patch`. Returns an existing +// repo entry, or a blank one — this is what lets a coordinator enrich a repo +// the pre-fetch never named at all (a brand-new candidate it found live), not +// just one that's already there with a gap. +function findRepoEntry(doc, repo) { + return doc.github[repo] ?? { deployState: null, prsInWindow: [], gap: null }; +} + +function findWorkloadEntry(doc, workload) { + return doc.logs[workload] ?? { clusterIds: [], kubectlSweep: null, victorialogs: null, gap: null }; +} + +/** Write back what a coordinator gathered LIVE for a repo — a deeper + * `deployState` (e.g. the full diff/patch, not just a summary), and/or one or + * more PRs to fold into `prsInWindow` (deduped by `pr`; a PR with a `pr` that + * already exists is REPLACED, since the coordinator's fresh finding is + * presumably deeper than a placeholder). `patch = { deployState?, prsInWindow?, + * gap? }` — omit a field to leave it untouched. Passing `gap: null` clears a + * previously-recorded gap now that real evidence exists. Never removes a PR + * or a field this call doesn't mention. */ +export function mergeGithubEvidence(filePath, repo, patch, nowMs) { + const doc = loadOrInit(filePath, nowMs); + const entry = findRepoEntry(doc, repo); + if (patch.deployState !== undefined) entry.deployState = patch.deployState; + if (Array.isArray(patch.prsInWindow)) { + const byPr = new Map((entry.prsInWindow ?? []).map((p) => [String(p.pr), p])); + for (const pr of patch.prsInWindow) byPr.set(String(pr.pr), pr); + entry.prsInWindow = [...byPr.values()]; + } + if (patch.gap !== undefined) entry.gap = patch.gap; + doc.github[repo] = entry; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return entry; +} + +/** Write back what a coordinator gathered LIVE for a workload's logs — same + * merge discipline as `mergeGithubEvidence`. `patch = { kubectlSweep?, + * victorialogs?, clusterIds?, gap? }`; `clusterIds` is unioned, not replaced, + * since more than one cluster can end up sharing a workload over the run. */ +export function mergeLogsEvidence(filePath, workload, patch, nowMs) { + const doc = loadOrInit(filePath, nowMs); + const entry = findWorkloadEntry(doc, workload); + if (patch.kubectlSweep !== undefined) entry.kubectlSweep = patch.kubectlSweep; + if (patch.victorialogs !== undefined) entry.victorialogs = patch.victorialogs; + if (Array.isArray(patch.clusterIds)) { + entry.clusterIds = [...new Set([...(entry.clusterIds ?? []), ...patch.clusterIds])]; + } + if (patch.gap !== undefined) entry.gap = patch.gap; + doc.logs[workload] = entry; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return entry; +} + // A requested item is "covered" only if it is present AND its own `gap` field // is falsy. Presence with a `gap` is a recorded, deliberate miss — not // coverage — so a coordinator (or this function) never mistakes "we looked diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 0f9f348..ca77c9b 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -336,6 +336,18 @@ representative's verdict just because the file already has the answer in it."* A dispatch prompt that omits this path forces its coordinator back into a full independent sweep — exactly the redundancy Step 4 exists to remove. +**The file is read-write, not just read-only.** When a coordinator has to +gather live (a genuine gap), tell it to write the result back — +`mergeGithubEvidence`/`mergeLogsEvidence` (`lib/evidence-file.mjs`) — before +finishing, not just answer TFA and move on. A representative's deep dive (a +full diff, a downstream trace, a PR the pre-fetch never named) then benefits +its own siblings and any other cluster sharing the same repo/workload, instead +of every one of them re-running the same live search. This is already baked +into `agents/ai-tfa-coordinator.md`'s Operating Principle 0 for any dispatch +of that agent type — no need to repeat the mechanics in the prompt, just don't +omit `evidenceFilePath` (above), since write-back has nothing to write to +without it. + **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index a01466a..768142b 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -12,6 +12,8 @@ import { setBaseline, setGithubEvidence, setLogsEvidence, + mergeGithubEvidence, + mergeLogsEvidence, recomputeCoverage, } from "../lib/evidence-file.mjs"; @@ -133,6 +135,75 @@ test("a block string with newlines and quotes round-trips through JSON unchanged assert.equal(doc.github["org/a"].deployState.block, block); }); +test("mergeGithubEvidence on a repo the pre-fetch never named creates a fresh entry", () => { + const entry = mergeGithubEvidence(file, "org/new-repo", { + prsInWindow: [{ pr: "#8912", verdict: "supported", block: "found live" }], + }, 1000); + assert.equal(entry.prsInWindow.length, 1); + assert.equal(entry.gap, null); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/new-repo"].prsInWindow[0].pr, "#8912"); +}); + +test("mergeGithubEvidence appends a new PR without dropping an existing one", () => { + setGithubEvidence(file, "org/a", { + gap: null, + deployState: { block: "a" }, + prsInWindow: [{ pr: "#1", verdict: "not-live" }], + }, 1000); + mergeGithubEvidence(file, "org/a", { + prsInWindow: [{ pr: "#2", verdict: "supported", block: "found live during coordinator's own hunt" }], + }, 2000); + const doc = readEvidenceFile(file); + const prs = doc.github["org/a"].prsInWindow.map((p) => p.pr); + assert.deepEqual(prs.sort(), ["#1", "#2"]); + assert.equal(doc.github["org/a"].deployState.block, "a"); // untouched +}); + +test("mergeGithubEvidence replaces a PR entry with the same pr number (deeper finding wins)", () => { + setGithubEvidence(file, "org/a", { + gap: null, + deployState: { block: "a" }, + prsInWindow: [{ pr: "#9011", verdict: "unassessed", files: null }], + }, 1000); + mergeGithubEvidence(file, "org/a", { + prsInWindow: [{ pr: "#9011", verdict: "supported", files: ["Foo.java"], block: "full diff fetched" }], + }, 2000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].prsInWindow.length, 1); + assert.equal(doc.github["org/a"].prsInWindow[0].verdict, "supported"); + assert.deepEqual(doc.github["org/a"].prsInWindow[0].files, ["Foo.java"]); +}); + +test("mergeGithubEvidence with gap:null clears a previously-recorded gap", () => { + setGithubEvidence(file, "org/a", { gap: "gh auth failed" }, 1000); + mergeGithubEvidence(file, "org/a", { gap: null, deployState: { block: "found it after all" } }, 2000); + const doc = readEvidenceFile(file); + assert.equal(doc.github["org/a"].gap, null); +}); + +test("mergeLogsEvidence unions clusterIds instead of replacing them", () => { + setLogsEvidence(file, "w1", { gap: null, clusterIds: ["c-A"], kubectlSweep: { block: "x" } }, 1000); + mergeLogsEvidence(file, "w1", { clusterIds: ["c-B"] }, 2000); + const doc = readEvidenceFile(file); + assert.deepEqual(doc.logs["w1"].clusterIds.sort(), ["c-A", "c-B"]); + assert.equal(doc.logs["w1"].kubectlSweep.block, "x"); // untouched +}); + +test("mergeLogsEvidence upgrades one sub-field without touching the other", () => { + setLogsEvidence(file, "w1", { + gap: null, + kubectlSweep: { gap: "stale pods" }, + victorialogs: { block: "clean, 0 5xx" }, + }, 1000); + mergeLogsEvidence(file, "w1", { + kubectlSweep: { block: "found a fresh pod after all, 3 matched lines" }, + }, 2000); + const doc = readEvidenceFile(file); + assert.equal(doc.logs["w1"].kubectlSweep.block, "found a fresh pod after all, 3 matched lines"); + assert.equal(doc.logs["w1"].victorialogs.block, "clean, 0 5xx"); // untouched +}); + test("writeEvidenceFile creates the parent directory if missing", () => { const nested = join(dir, "nested", "sub", "evidence.json"); writeEvidenceFile(nested, emptyEvidenceFile("build-1", 0)); diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 1624b54..d7d527f 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -67,6 +67,7 @@ const shared = [ `Pre-fetched build-evidence file — READ THIS FIRST (via the Read tool) before making ANY live github/infra/logs gather call: ${ctx.evidenceFilePath}`, `Build-evidence summary (full detail is in the file above; this is only a pointer — do not re-fetch what the file already covers): ${JSON.stringify(ctx.buildEvidence ?? {})}`, `If the file's github/logs sections do not name a repo/workload/ask you need, or record a "gap" for it, that is a genuine gap — fall back to a live gather via the capability manifest above exactly as if no file existed. The file is an optimization, never a hard dependency.`, + `The file is read-write: after any live gather that fills a gap or goes deeper than what was there, write it back via mergeGithubEvidence/mergeLogsEvidence (lib/evidence-file.mjs) before finishing this test, so a sibling dispatched after you — or another cluster sharing the same repo/workload — reads the enriched entry instead of re-fetching it.`, `Autonomous run — on an evidence gap with no valid connector, report "unavailable" back to TFA (NEVER prompt a user). Best-effort finalize.`, `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, From d33e99eaafb8f846973c2151c2548ef37ebc3480 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 16:21:52 +0530 Subject: [PATCH 09/51] fix(rca): eliminate lost evidence write-backs via single-writer shards The write-back added in 70ef859 had every coordinator read-modify-writing one shared JSON file. Measured under 8 concurrent writers with a realistic read -> work -> write window, that design lost 28 of 40 updates (70%). Replace it with a layout where contention is structurally impossible instead of merely unlikely: the orchestrator solely owns the base file, and each coordinator writes only its own shard at rca-evidence.<buildId>.contrib/<testRunId>.json. No two processes ever open the same file for writing. readEvidenceFile folds base + all shards into one view (sorted order; real evidence beats a recorded gap; PRs unioned by number), and readBaseFile keeps the orchestrator's own write path from absorbing shard content back into base. Same 8-writer test against the new layout: 0 of 40 lost. A 12-process run hammering one repo key kept all 480 writes. A corrupt/half-written shard is skipped rather than breaking every subsequent read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 34 +++-- lib/evidence-file.mjs | 245 +++++++++++++++++++++++++++-------- skills/rca-build/SKILL.md | 26 +++- tests/evidence-file.test.mjs | 147 +++++++++++++-------- workflows/rca-batch.mjs | 2 +- 5 files changed, 320 insertions(+), 134 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 9a43df1..80c7668 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -46,8 +46,9 @@ it names no `kubectl` / `chitragupta` / `bifrost`; it routes by *capability*. for every repo/workload this build's failures implicate. `Read` it before any live gather call (see Operating Principle 0) — and treat it as read-WRITE: a live gather that fills a gap or goes deeper is written back - via `mergeGithubEvidence`/`mergeLogsEvidence` so later dispatches (this - test's own siblings, or another cluster sharing the same repo/workload) + via `contributeGithubEvidence`/`contributeLogsEvidence` (writing your own + per-writer shard, keyed by your `testRunId`) so later dispatches — this + test's own siblings, or another cluster sharing the same repo/workload — benefit too. If `testRunId` is missing or not parseable as an integer, emit a `failed` @@ -94,15 +95,19 @@ read-only and has no side effects, so a read is always safe to repeat. the pre-fetch never named, a log sweep that succeeded where the file recorded one as gapped) is exactly the kind of build-level fact this file exists to share — not just this test's own answer. Persist it via - `mergeGithubEvidence(evidenceFilePath, repo, patch, nowMs)` or - `mergeLogsEvidence(evidenceFilePath, workload, patch, nowMs)` - (`lib/evidence-file.mjs`) before finishing this test, so a sibling - dispatched after you (or any other cluster that turns out to share the - same repo/workload) reads the enriched entry instead of re-fetching what - you just fetched. Only write back genuinely new/deeper findings — don't - write back a no-op read of an already-covered entry. This has the same - informal-locking caveat as the CSV: it's a best-effort optimization, not a - correctness dependency, so never block or retry on it. + `contributeGithubEvidence(evidenceFilePath, writerId, repo, patch, nowMs)` + or `contributeLogsEvidence(evidenceFilePath, writerId, workload, patch, + nowMs)` (`lib/evidence-file.mjs`), where **`writerId` is your own + `testRunId`** — that is what keeps writes safe. Each coordinator writes only + its own shard file under `<evidenceFilePath minus .json>.contrib/`, so + concurrent coordinators can never clobber each other or the orchestrator's + base pre-fetch; readers fold base + every shard back into one view + automatically. Write back before finishing this test, so a sibling + dispatched after you (or any other cluster sharing the same repo/workload) + reads the enriched entry instead of re-fetching what you just fetched. + Only write back genuinely new/deeper findings — never a no-op re-write of + an already-covered entry. It's a best-effort optimization, not a + correctness dependency: never block or retry on it. 1. **Logs by TFA — the core contract.** Never seed logs in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or digest log content. Logs are TFA's job. @@ -182,8 +187,8 @@ re-fetch per test (the `evidenceFile`'s `github` section, if present and not `gap`-marked for this repo; otherwise the live github connector). A culprit hunt often needs to go deeper than the file's summary — a full diff, a downstream consumer of a changed flag — write that depth back via -`mergeGithubEvidence` once found, so a sibling confirming the same suspect PR -doesn't re-run the same diff/search. Never fabricate a PR when the github +`contributeGithubEvidence` once found, so a sibling confirming the same +suspect PR doesn't re-run the same diff/search. Never fabricate a PR when the github capability is unavailable — emit an `unavailable` block. @@ -218,7 +223,8 @@ capability is unavailable — emit an re-digesting, no live call. Not named in the file, or its entry has a `gap`, or no `evidenceFile` at all → run the discovered skill/tool live, exactly as before — THEN write the - result back via `mergeGithubEvidence`/`mergeLogsEvidence` + result back via `contributeGithubEvidence`/ + `contributeLogsEvidence` with your own testRunId as writerId (Operating Principle 0) so this fills the gap for whoever reads the file next. Digest into one block. Record evidenceType in asks_fulfilled (dedupe). diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index d7487e5..b9010b6 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -25,34 +25,85 @@ // discipline as `csv-state.mjs`, so this stays usable from the Workflow-tool // sandbox (which forbids `Date.now()`). // -// Write-back (mergeGithubEvidence / mergeLogsEvidence): the orchestrator's -// Step 4 pass is not the only writer. A coordinator that had to gather live -// (a repo/PR/workload the pre-fetch didn't cover, or covered only with a -// summary) should write what it found back into this SAME file — a -// representative's deep dive then benefits its own siblings (dispatched -// after it resolves) and any other cluster that turns out to share the same -// repo/workload, without re-fetching. Concurrency caveat, same philosophy as -// `csv-state.mjs`'s "true multi-process locking is out of scope": this is a -// synchronous read-modify-write with no file lock, so two coordinators -// writing to the SAME repo/workload key at truly the same moment can lose an -// update. In practice this is low-risk for the case this exists to serve — -// a sibling is dispatched only after its representative resolves, i.e. -// sequentially, never concurrently with it — and acceptable for the rarer -// case of two parallel representatives happening to touch the same repo. - -import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs"; +// Write-back, WITHOUT a lock and WITHOUT lost updates — single-writer shards. +// The orchestrator's Step 4 pass is not the only writer: a coordinator that +// had to gather live (a repo/PR/workload the pre-fetch didn't cover, or +// covered only with a summary) should persist what it found so a sibling +// dispatched after it — or another cluster sharing the same repo/workload — +// reads the enriched result instead of re-fetching it. +// +// Naively that means N concurrent coordinators read-modify-writing ONE JSON +// file, which drops updates whenever two writes interleave. Instead of a lock +// (fragile, and `csv-state.mjs` already declares multi-process locking out of +// scope) the layout makes contention structurally impossible: +// +// <tmpdir>/bstack-rca/ +// rca-evidence.<buildId>.json <- BASE: only the orchestrator writes it +// rca-evidence.<buildId>.contrib/ +// <writerId>.json <- one file per coordinator; sole writer +// <writerId>.json +// +// Every file has exactly ONE writer, so no write can ever clobber another's. +// Reads (`readEvidenceFile`) fold base + every shard into a single view, +// deterministically (shards applied in sorted filename order). This is the +// same "the temp dir is ours, use more of it" trick the CSV path convention +// already leans on. + +// Fold precedence, applied per leaf when base and shards disagree: +// 1. Real evidence beats a recorded gap — a coordinator that actually got +// the data overrides the pre-fetch's "couldn't reach this". +// 2. Among two real values, the later shard wins (sorted order), on the +// assumption a coordinator only writes back something deeper than what +// it read. +// 3. `prsInWindow` is unioned by PR number rather than replaced, so two +// coordinators finding different PRs in the same repo both survive. + +import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync } from "node:fs"; import { dirname, join } from "node:path"; import { tmpdir } from "node:os"; +const safeName = (v, fallback) => + String(v ?? "").replace(/[^A-Za-z0-9._-]/g, "_") || fallback; + /** Canonical evidence-file path for one build's run. Same two invariants as * `csvPathFor`: build id in the filename; OS temp by default; `stateDir` * overrides the directory only. */ export function evidencePathFor(buildId, stateDir = "") { - const safe = String(buildId ?? "").replace(/[^A-Za-z0-9._-]/g, "_") || "unknown-build"; + const safe = safeName(buildId, "unknown-build"); const dir = stateDir && String(stateDir).trim() !== "" ? String(stateDir) : join(tmpdir(), "bstack-rca"); return join(dir, `rca-evidence.${safe}.json`); } +/** Directory holding this build's per-coordinator contribution shards. Derived + * from the base path so callers only ever have to pass `evidenceFilePath` + * around — one input, no second path to thread through every dispatch. */ +export function contribDirFor(basePath) { + return String(basePath).replace(/\.json$/, "") + ".contrib"; +} + +/** The one file a given writer owns. Exactly one writer per path is the whole + * point — never call this for a writerId that isn't yours. */ +export function contribPathFor(basePath, writerId) { + return join(contribDirFor(basePath), `${safeName(writerId, "unknown-writer")}.json`); +} + +/** Shard docs in deterministic (sorted-filename) order. Missing dir → []. A + * corrupt/half-written shard is skipped rather than throwing: a coordinator + * killed mid-write must not break every subsequent read. */ +function readContribs(basePath) { + const dir = contribDirFor(basePath); + if (!existsSync(dir)) return []; + const out = []; + for (const name of readdirSync(dir).filter((n) => n.endsWith(".json")).sort()) { + try { + out.push(JSON.parse(readFileSync(join(dir, name), "utf8"))); + } catch { + // skip unreadable/partial shard + } + } + return out; +} + export function emptyEvidenceFile(buildId, nowMs) { return { buildId: String(buildId ?? ""), @@ -65,13 +116,78 @@ export function emptyEvidenceFile(buildId, nowMs) { }; } -/** Read-only; never throws on a missing file — a coordinator (or the - * orchestrator, before Step 4 has run) always gets a well-shaped, empty-covered - * result rather than an exception. Graceful degradation is the point: an - * absent/partial file just means every ask falls back to a live gather. */ -export function readEvidenceFile(filePath) { +/** The BASE file alone, no shards folded in. Internal to the orchestrator's + * write path: `set*`/`merge*` must read-modify-write base only, or they would + * absorb shard content into base and duplicate it on the next fold. */ +export function readBaseFile(filePath) { if (!existsSync(filePath)) return emptyEvidenceFile("unknown-build", 0); - return JSON.parse(readFileSync(filePath, "utf8")); + try { + return JSON.parse(readFileSync(filePath, "utf8")); + } catch { + return emptyEvidenceFile("unknown-build", 0); + } +} + +// A {block, gap} leaf: real evidence beats a gap; between two real values the +// later (shard) one wins. `undefined`/`null` incoming never overwrites. +function pickLeaf(base, incoming) { + if (incoming == null) return base ?? null; + if (base == null) return incoming; + if (!incoming.gap) return incoming; + if (!base.gap) return base; + return incoming; +} + +function foldGithub(target, repo, entry) { + const cur = target[repo] ?? { deployState: null, prsInWindow: [], gap: null }; + const next = { + deployState: pickLeaf(cur.deployState, entry.deployState), + prsInWindow: cur.prsInWindow ?? [], + gap: cur.gap ?? null, + }; + if (Array.isArray(entry.prsInWindow)) { + const byPr = new Map((next.prsInWindow ?? []).map((p) => [String(p.pr), p])); + for (const pr of entry.prsInWindow) byPr.set(String(pr.pr), pr); + next.prsInWindow = [...byPr.values()]; + } + // A contributor supplying real content clears the pre-fetch's gap. + if (entry.gap === null || entry.gap === undefined) { + if (entry.deployState || Array.isArray(entry.prsInWindow)) next.gap = null; + } else if (!next.deployState && (next.prsInWindow ?? []).length === 0) { + next.gap = entry.gap; + } + target[repo] = next; +} + +function foldLogs(target, workload, entry) { + const cur = target[workload] ?? { clusterIds: [], kubectlSweep: null, victorialogs: null, gap: null }; + const next = { + clusterIds: [...new Set([...(cur.clusterIds ?? []), ...(entry.clusterIds ?? [])])], + kubectlSweep: pickLeaf(cur.kubectlSweep, entry.kubectlSweep), + victorialogs: pickLeaf(cur.victorialogs, entry.victorialogs), + gap: cur.gap ?? null, + }; + if (entry.gap === null || entry.gap === undefined) { + if (entry.kubectlSweep || entry.victorialogs) next.gap = null; + } else if (!next.kubectlSweep && !next.victorialogs) { + next.gap = entry.gap; + } + target[workload] = next; +} + +/** The full view every CONSUMER should read: the orchestrator's base pre-fetch + * with every coordinator's contribution shard folded on top, deterministically. + * Never throws on a missing base or a corrupt shard — an absent/partial result + * just means those asks fall back to a live gather, which is the whole + * degradation contract. */ +export function readEvidenceFile(filePath) { + const doc = readBaseFile(filePath); + for (const shard of readContribs(filePath)) { + for (const [repo, entry] of Object.entries(shard.github ?? {})) foldGithub(doc.github, repo, entry); + for (const [wl, entry] of Object.entries(shard.logs ?? {})) foldLogs(doc.logs, wl, entry); + if (shard.generatedAtMs > (doc.generatedAtMs ?? 0)) doc.generatedAtMs = shard.generatedAtMs; + } + return doc; } export function writeEvidenceFile(filePath, doc) { @@ -82,7 +198,7 @@ export function writeEvidenceFile(filePath, doc) { function loadOrInit(filePath, nowMs) { if (!existsSync(filePath)) return emptyEvidenceFile("unknown-build", nowMs); - return readEvidenceFile(filePath); + return readBaseFile(filePath); } /** Idempotent: creates the file with the given `buildId` if it doesn't exist @@ -91,7 +207,7 @@ function loadOrInit(filePath, nowMs) { * is recorded correctly — the `set*` functions below fall back to * `"unknown-build"` only as a safety net if called without this. */ export function initEvidenceFile(filePath, buildId, nowMs) { - if (existsSync(filePath)) return readEvidenceFile(filePath); + if (existsSync(filePath)) return readBaseFile(filePath); const doc = emptyEvidenceFile(buildId, nowMs); writeEvidenceFile(filePath, doc); return doc; @@ -136,29 +252,41 @@ export function setLogsEvidence(filePath, workload, entry, nowMs) { return doc; } -// Read-modify-write, preserving whatever isn't in `patch`. Returns an existing -// repo entry, or a blank one — this is what lets a coordinator enrich a repo -// the pre-fetch never named at all (a brand-new candidate it found live), not -// just one that's already there with a gap. -function findRepoEntry(doc, repo) { - return doc.github[repo] ?? { deployState: null, prsInWindow: [], gap: null }; +// ---- coordinator write-back: own-shard only, never the base file ---------- +// +// `writerId` must be unique per concurrent writer — the dispatched +// coordinator's `testRunId` is the natural choice (one coordinator per test). +// Because a writer only ever opens its OWN shard, two coordinators writing at +// the same instant touch different files and neither can lose the other's +// update. Reads fold every shard back together (`readEvidenceFile`). + +function loadOwnShard(basePath, writerId, nowMs) { + const p = contribPathFor(basePath, writerId); + if (!existsSync(p)) { + return { path: p, doc: { writerId: String(writerId), generatedAtMs: nowMs, github: {}, logs: {} } }; + } + try { + return { path: p, doc: JSON.parse(readFileSync(p, "utf8")) }; + } catch { + return { path: p, doc: { writerId: String(writerId), generatedAtMs: nowMs, github: {}, logs: {} } }; + } } -function findWorkloadEntry(doc, workload) { - return doc.logs[workload] ?? { clusterIds: [], kubectlSweep: null, victorialogs: null, gap: null }; +function writeShard(path, doc) { + const dir = dirname(path); + if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true }); + writeFileSync(path, JSON.stringify(doc, null, 2), "utf8"); } -/** Write back what a coordinator gathered LIVE for a repo — a deeper - * `deployState` (e.g. the full diff/patch, not just a summary), and/or one or - * more PRs to fold into `prsInWindow` (deduped by `pr`; a PR with a `pr` that - * already exists is REPLACED, since the coordinator's fresh finding is - * presumably deeper than a placeholder). `patch = { deployState?, prsInWindow?, - * gap? }` — omit a field to leave it untouched. Passing `gap: null` clears a - * previously-recorded gap now that real evidence exists. Never removes a PR - * or a field this call doesn't mention. */ -export function mergeGithubEvidence(filePath, repo, patch, nowMs) { - const doc = loadOrInit(filePath, nowMs); - const entry = findRepoEntry(doc, repo); +/** Contribute what THIS coordinator gathered live for a repo — a deeper + * `deployState` (e.g. the full diff, not just a summary), and/or PRs to fold + * into `prsInWindow` (deduped by `pr`). `patch = { deployState?, prsInWindow?, + * gap? }`; omit a field to leave it untouched. Writes only this writer's + * shard, so it can never clobber another coordinator's contribution or the + * orchestrator's base pre-fetch. */ +export function contributeGithubEvidence(basePath, writerId, repo, patch, nowMs) { + const { path, doc } = loadOwnShard(basePath, writerId, nowMs); + const entry = doc.github[repo] ?? { deployState: null, prsInWindow: [], gap: null }; if (patch.deployState !== undefined) entry.deployState = patch.deployState; if (Array.isArray(patch.prsInWindow)) { const byPr = new Map((entry.prsInWindow ?? []).map((p) => [String(p.pr), p])); @@ -168,17 +296,16 @@ export function mergeGithubEvidence(filePath, repo, patch, nowMs) { if (patch.gap !== undefined) entry.gap = patch.gap; doc.github[repo] = entry; doc.generatedAtMs = nowMs; - writeEvidenceFile(filePath, doc); + writeShard(path, doc); return entry; } -/** Write back what a coordinator gathered LIVE for a workload's logs — same - * merge discipline as `mergeGithubEvidence`. `patch = { kubectlSweep?, - * victorialogs?, clusterIds?, gap? }`; `clusterIds` is unioned, not replaced, - * since more than one cluster can end up sharing a workload over the run. */ -export function mergeLogsEvidence(filePath, workload, patch, nowMs) { - const doc = loadOrInit(filePath, nowMs); - const entry = findWorkloadEntry(doc, workload); +/** Contribute what THIS coordinator gathered live for a workload's app-logs. + * Same single-writer-shard discipline as `contributeGithubEvidence`; + * `clusterIds` is unioned rather than replaced. */ +export function contributeLogsEvidence(basePath, writerId, workload, patch, nowMs) { + const { path, doc } = loadOwnShard(basePath, writerId, nowMs); + const entry = doc.logs[workload] ?? { clusterIds: [], kubectlSweep: null, victorialogs: null, gap: null }; if (patch.kubectlSweep !== undefined) entry.kubectlSweep = patch.kubectlSweep; if (patch.victorialogs !== undefined) entry.victorialogs = patch.victorialogs; if (Array.isArray(patch.clusterIds)) { @@ -187,7 +314,7 @@ export function mergeLogsEvidence(filePath, workload, patch, nowMs) { if (patch.gap !== undefined) entry.gap = patch.gap; doc.logs[workload] = entry; doc.generatedAtMs = nowMs; - writeEvidenceFile(filePath, doc); + writeShard(path, doc); return entry; } @@ -205,14 +332,18 @@ function isCovered(doc, section, key) { * — normally the Gate Part A scope-probe-validated repo list and the union of * workloads every cluster's representative implicates (see SKILL.md Step 4). */ export function recomputeCoverage(filePath, requested, nowMs) { + // Coverage is judged against the FOLDED view — a gap the orchestrator + // recorded but a coordinator later filled is genuinely covered now — while + // the result is persisted to base, which the orchestrator solely owns. + const folded = readEvidenceFile(filePath); const doc = loadOrInit(filePath, nowMs); const repos = requested?.repos ?? []; const workloads = requested?.workloads ?? []; doc.coverage = { - reposCovered: repos.filter((r) => isCovered(doc, "github", r)), - reposGapped: repos.filter((r) => !isCovered(doc, "github", r)), - workloadsCovered: workloads.filter((w) => isCovered(doc, "logs", w)), - workloadsGapped: workloads.filter((w) => !isCovered(doc, "logs", w)), + reposCovered: repos.filter((r) => isCovered(folded, "github", r)), + reposGapped: repos.filter((r) => !isCovered(folded, "github", r)), + workloadsCovered: workloads.filter((w) => isCovered(folded, "logs", w)), + workloadsGapped: workloads.filter((w) => !isCovered(folded, "logs", w)), }; doc.generatedAtMs = nowMs; writeEvidenceFile(filePath, doc); diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index ca77c9b..686441d 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -338,16 +338,28 @@ a full independent sweep — exactly the redundancy Step 4 exists to remove. **The file is read-write, not just read-only.** When a coordinator has to gather live (a genuine gap), tell it to write the result back — -`mergeGithubEvidence`/`mergeLogsEvidence` (`lib/evidence-file.mjs`) — before -finishing, not just answer TFA and move on. A representative's deep dive (a -full diff, a downstream trace, a PR the pre-fetch never named) then benefits -its own siblings and any other cluster sharing the same repo/workload, instead -of every one of them re-running the same live search. This is already baked -into `agents/ai-tfa-coordinator.md`'s Operating Principle 0 for any dispatch -of that agent type — no need to repeat the mechanics in the prompt, just don't +`contributeGithubEvidence`/`contributeLogsEvidence` (`lib/evidence-file.mjs`), +passing its own `testRunId` as `writerId` — before finishing, not just answer +TFA and move on. A representative's deep dive (a full diff, a downstream +trace, a PR the pre-fetch never named) then benefits its own siblings and any +other cluster sharing the same repo/workload, instead of every one of them +re-running the same live search. This is already baked into +`agents/ai-tfa-coordinator.md`'s Operating Principle 0 for any dispatch of +that agent type — no need to repeat the mechanics in the prompt, just don't omit `evidenceFilePath` (above), since write-back has nothing to write to without it. +**Concurrency is handled by layout, not by locking.** Base +(`rca-evidence.<buildId>.json`) has exactly one writer — this orchestrator, in +Step 4. Every coordinator writes only its own shard under +`rca-evidence.<buildId>.contrib/<testRunId>.json`. Since no two processes ever +open the same file for writing, concurrent write-back cannot lose an update; +`readEvidenceFile` folds base + all shards into one view, applying shards in +sorted order, with real evidence taking precedence over a recorded `gap`. A +measured comparison: 8 concurrent writers with a realistic read→work→write +window lost **28 of 40 updates** against a single shared file, and **0 of 40** +under this layout. + **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index 768142b..dc40742 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -1,6 +1,6 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -12,8 +12,11 @@ import { setBaseline, setGithubEvidence, setLogsEvidence, - mergeGithubEvidence, - mergeLogsEvidence, + contributeGithubEvidence, + contributeLogsEvidence, + contribDirFor, + contribPathFor, + readBaseFile, recomputeCoverage, } from "../lib/evidence-file.mjs"; @@ -135,73 +138,107 @@ test("a block string with newlines and quotes round-trips through JSON unchanged assert.equal(doc.github["org/a"].deployState.block, block); }); -test("mergeGithubEvidence on a repo the pre-fetch never named creates a fresh entry", () => { - const entry = mergeGithubEvidence(file, "org/new-repo", { - prsInWindow: [{ pr: "#8912", verdict: "supported", block: "found live" }], - }, 1000); - assert.equal(entry.prsInWindow.length, 1); - assert.equal(entry.gap, null); - const doc = readEvidenceFile(file); - assert.equal(doc.github["org/new-repo"].prsInWindow[0].pr, "#8912"); +test("contribute writes a shard, never the base file", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "base" } }, 1000); + contributeGithubEvidence(file, "3895581484", "org/a", { + deployState: { block: "coordinator's full diff" }, + }, 2000); + // base is untouched... + assert.equal(readBaseFile(file).github["org/a"].deployState.block, "base"); + // ...but the folded view shows the contribution + assert.equal(readEvidenceFile(file).github["org/a"].deployState.block, "coordinator's full diff"); }); -test("mergeGithubEvidence appends a new PR without dropping an existing one", () => { - setGithubEvidence(file, "org/a", { - gap: null, - deployState: { block: "a" }, - prsInWindow: [{ pr: "#1", verdict: "not-live" }], - }, 1000); - mergeGithubEvidence(file, "org/a", { - prsInWindow: [{ pr: "#2", verdict: "supported", block: "found live during coordinator's own hunt" }], - }, 2000); - const doc = readEvidenceFile(file); - const prs = doc.github["org/a"].prsInWindow.map((p) => p.pr); - assert.deepEqual(prs.sort(), ["#1", "#2"]); - assert.equal(doc.github["org/a"].deployState.block, "a"); // untouched +test("contribPathFor: one file per writer, under the build's contrib dir", () => { + const p = contribPathFor(file, "3895581484"); + assert.ok(p.startsWith(contribDirFor(file))); + assert.ok(p.endsWith("3895581484.json")); + assert.notEqual(contribPathFor(file, "w1"), contribPathFor(file, "w2")); +}); + +test("contribPathFor sanitizes a hostile writerId", () => { + assert.ok(contribPathFor(file, "../../etc/passwd").endsWith("_.._etc_passwd.json")); }); -test("mergeGithubEvidence replaces a PR entry with the same pr number (deeper finding wins)", () => { +test("CONCURRENCY: two writers on the same repo both survive (no lost update)", () => { setGithubEvidence(file, "org/a", { - gap: null, - deployState: { block: "a" }, - prsInWindow: [{ pr: "#9011", verdict: "unassessed", files: null }], + gap: null, deployState: { block: "base" }, prsInWindow: [{ pr: "#1" }], }, 1000); - mergeGithubEvidence(file, "org/a", { - prsInWindow: [{ pr: "#9011", verdict: "supported", files: ["Foo.java"], block: "full diff fetched" }], - }, 2000); - const doc = readEvidenceFile(file); - assert.equal(doc.github["org/a"].prsInWindow.length, 1); - assert.equal(doc.github["org/a"].prsInWindow[0].verdict, "supported"); - assert.deepEqual(doc.github["org/a"].prsInWindow[0].files, ["Foo.java"]); + // Interleave the two writers the way real concurrent coordinators would: + // each reads, then each writes — under a single shared file this is exactly + // the sequence that drops the first writer's update. + contributeGithubEvidence(file, "writerA", "org/a", { prsInWindow: [{ pr: "#2", by: "A" }] }, 2000); + contributeGithubEvidence(file, "writerB", "org/a", { prsInWindow: [{ pr: "#3", by: "B" }] }, 2000); + const prs = readEvidenceFile(file).github["org/a"].prsInWindow.map((p) => p.pr).sort(); + assert.deepEqual(prs, ["#1", "#2", "#3"]); // base + BOTH contributions +}); + +test("CONCURRENCY: two writers on the same workload both survive", () => { + contributeLogsEvidence(file, "writerA", "w1", { kubectlSweep: { block: "A found 3 lines" } }, 1000); + contributeLogsEvidence(file, "writerB", "w1", { victorialogs: { block: "B found 5xx" } }, 1000); + const w = readEvidenceFile(file).logs["w1"]; + assert.equal(w.kubectlSweep.block, "A found 3 lines"); + assert.equal(w.victorialogs.block, "B found 5xx"); }); -test("mergeGithubEvidence with gap:null clears a previously-recorded gap", () => { +test("fold: real contributed evidence beats a base-recorded gap", () => { setGithubEvidence(file, "org/a", { gap: "gh auth failed" }, 1000); - mergeGithubEvidence(file, "org/a", { gap: null, deployState: { block: "found it after all" } }, 2000); - const doc = readEvidenceFile(file); - assert.equal(doc.github["org/a"].gap, null); + contributeGithubEvidence(file, "w1", "org/a", { + gap: null, deployState: { block: "reachable after all" }, + }, 2000); + const entry = readEvidenceFile(file).github["org/a"]; + assert.equal(entry.gap, null); + assert.equal(entry.deployState.block, "reachable after all"); }); -test("mergeLogsEvidence unions clusterIds instead of replacing them", () => { - setLogsEvidence(file, "w1", { gap: null, clusterIds: ["c-A"], kubectlSweep: { block: "x" } }, 1000); - mergeLogsEvidence(file, "w1", { clusterIds: ["c-B"] }, 2000); - const doc = readEvidenceFile(file); - assert.deepEqual(doc.logs["w1"].clusterIds.sort(), ["c-A", "c-B"]); - assert.equal(doc.logs["w1"].kubectlSweep.block, "x"); // untouched +test("fold: a contributed gap does NOT overwrite real base evidence", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "real base evidence" } }, 1000); + contributeGithubEvidence(file, "w1", "org/a", { deployState: { gap: "my call failed" } }, 2000); + assert.equal(readEvidenceFile(file).github["org/a"].deployState.block, "real base evidence"); }); -test("mergeLogsEvidence upgrades one sub-field without touching the other", () => { - setLogsEvidence(file, "w1", { - gap: null, - kubectlSweep: { gap: "stale pods" }, - victorialogs: { block: "clean, 0 5xx" }, +test("fold: same PR number contributed later wins (deeper finding replaces placeholder)", () => { + setGithubEvidence(file, "org/a", { + gap: null, prsInWindow: [{ pr: "#9011", verdict: "unassessed", files: null }], }, 1000); - mergeLogsEvidence(file, "w1", { - kubectlSweep: { block: "found a fresh pod after all, 3 matched lines" }, + contributeGithubEvidence(file, "w1", "org/a", { + prsInWindow: [{ pr: "#9011", verdict: "supported", files: ["Foo.java"] }], }, 2000); - const doc = readEvidenceFile(file); - assert.equal(doc.logs["w1"].kubectlSweep.block, "found a fresh pod after all, 3 matched lines"); - assert.equal(doc.logs["w1"].victorialogs.block, "clean, 0 5xx"); // untouched + const prs = readEvidenceFile(file).github["org/a"].prsInWindow; + assert.equal(prs.length, 1); + assert.equal(prs[0].verdict, "supported"); +}); + +test("fold: contributing a repo the pre-fetch never named", () => { + contributeGithubEvidence(file, "w1", "org/brand-new", { + prsInWindow: [{ pr: "#8912", verdict: "supported" }], + }, 1000); + assert.equal(readEvidenceFile(file).github["org/brand-new"].prsInWindow[0].pr, "#8912"); +}); + +test("fold: clusterIds union across base and multiple shards", () => { + setLogsEvidence(file, "w1", { gap: null, clusterIds: ["c-A"], kubectlSweep: { block: "x" } }, 1000); + contributeLogsEvidence(file, "w1writer", "w1", { clusterIds: ["c-B"] }, 2000); + contributeLogsEvidence(file, "w2writer", "w1", { clusterIds: ["c-C"] }, 2000); + assert.deepEqual(readEvidenceFile(file).logs["w1"].clusterIds.sort(), ["c-A", "c-B", "c-C"]); +}); + +test("fold: a corrupt shard is skipped, not fatal", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "base" } }, 1000); + contributeGithubEvidence(file, "good", "org/a", { prsInWindow: [{ pr: "#2" }] }, 2000); + writeFileSync(contribPathFor(file, "corrupt"), "{not json", "utf8"); + const doc = readEvidenceFile(file); // must not throw + assert.equal(doc.github["org/a"].prsInWindow[0].pr, "#2"); +}); + +test("recomputeCoverage counts a coordinator-filled gap as covered", () => { + setGithubEvidence(file, "org/a", { gap: "unreachable at pre-fetch time" }, 1000); + let cov = recomputeCoverage(file, { repos: ["org/a"], workloads: [] }, 2000); + assert.deepEqual(cov.reposGapped, ["org/a"]); + contributeGithubEvidence(file, "w1", "org/a", { gap: null, deployState: { block: "got it" } }, 3000); + cov = recomputeCoverage(file, { repos: ["org/a"], workloads: [] }, 4000); + assert.deepEqual(cov.reposCovered, ["org/a"]); + assert.deepEqual(cov.reposGapped, []); }); test("writeEvidenceFile creates the parent directory if missing", () => { diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index d7d527f..2a7a0e4 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -67,7 +67,7 @@ const shared = [ `Pre-fetched build-evidence file — READ THIS FIRST (via the Read tool) before making ANY live github/infra/logs gather call: ${ctx.evidenceFilePath}`, `Build-evidence summary (full detail is in the file above; this is only a pointer — do not re-fetch what the file already covers): ${JSON.stringify(ctx.buildEvidence ?? {})}`, `If the file's github/logs sections do not name a repo/workload/ask you need, or record a "gap" for it, that is a genuine gap — fall back to a live gather via the capability manifest above exactly as if no file existed. The file is an optimization, never a hard dependency.`, - `The file is read-write: after any live gather that fills a gap or goes deeper than what was there, write it back via mergeGithubEvidence/mergeLogsEvidence (lib/evidence-file.mjs) before finishing this test, so a sibling dispatched after you — or another cluster sharing the same repo/workload — reads the enriched entry instead of re-fetching it.`, + `The file is read-write: after any live gather that fills a gap or goes deeper than what was there, write it back via contributeGithubEvidence/contributeLogsEvidence (lib/evidence-file.mjs) passing your own testRunId as writerId, before finishing this test — so a sibling dispatched after you, or another cluster sharing the same repo/workload, reads the enriched entry instead of re-fetching it. Each writer owns its own shard file, so concurrent coordinators cannot clobber each other; readers fold base + shards automatically.`, `Autonomous run — on an evidence gap with no valid connector, report "unavailable" back to TFA (NEVER prompt a user). Best-effort finalize.`, `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, From 8becf43e32cf3d422b54fe7ce2ea00df7b2cfd90 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 16:41:01 +0530 Subject: [PATCH 10/51] feat(rca): memoize read-only tool calls; fast-fail wedged TFA drains MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two measured sources of duplicate work on a real 10-test build: 1. `gh` was 37% of all coordinator tool calls (151), and 46 were byte-identical commands re-run by different coordinators — one BStackAutomation spec fetched 12 times, a frontend component 5 times. None of these fit the evidence file's schema (deployState / prsInWindow), so the write-back added earlier could not share them. Add a build-scoped memo cache keyed by the CALL rather than by evidence shape, so duplication is caught regardless of what the call was for. `bin/cached-exec.mjs` wraps a shell fetch transparently (same stdout, same exit code, executes only on a miss); `bin/cached-mcp.mjs` does check-then- store for read-only MCP queries. One file per call key + atomic rename, so concurrent writers cannot collide or produce a torn read. Failures are never cached (a transient rate-limit must not become a permanent answer), stateful tools (tfaRcaTurn/getTfaTurnResult/triggerRcaReport) are refused, secrets are redacted before anything reaches disk, and the runner uses execFile with a hand-rolled tokenizer so no shell ever interprets an argument. 2. Drain reads plus their sleeps were 23% of all coordinator tool calls. The drain treated "TFA is still thinking" and "the TFA run hard-failed" identically, spending the full 40-read/10-min budget on turns that had already died — the four tests that wedged this way were the four slowest in the batch. Distinguish the two: stop after `maxErrorReads` (default 3) CONSECUTIVE hard failures, note it as `tfa-error`, keep the row resumable. A single good read clears the streak, so flaky-but-recovering reads still land. Measured: cross-writer cache hit served in 0.185s vs 1.131s for the live fetch. 108 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 28 +++++ bin/cached-exec.mjs | 93 +++++++++++++++ bin/cached-mcp.mjs | 98 ++++++++++++++++ config/rca.config.json | 4 +- lib/loop.mjs | 67 +++++++++-- lib/tool-cache.mjs | 222 +++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 11 ++ tests/conformance.test.mjs | 61 ++++++++++ tests/tool-cache.test.mjs | 151 ++++++++++++++++++++++++ workflows/rca-batch.mjs | 4 +- 10 files changed, 724 insertions(+), 15 deletions(-) create mode 100644 bin/cached-exec.mjs create mode 100644 bin/cached-mcp.mjs create mode 100644 lib/tool-cache.mjs create mode 100644 tests/tool-cache.test.mjs diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 80c7668..1ae0096 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -108,6 +108,34 @@ read-only and has no side effects, so a read is always safe to repeat. Only write back genuinely new/deeper findings — never a no-op re-write of an already-covered entry. It's a best-effort optimization, not a correctness dependency: never block or retry on it. + + **Route read-only lookups through the tool cache.** The evidence file + shares *digested findings*; the cache below shares *raw call results*, which + is where most duplicate work actually hides (measured on one real build: + `gh` was 37% of all coordinator tool calls, 46 of them byte-identical + commands re-run by different coordinators — one spec file fetched 12 + times). Given `buildId` and your own `testRunId` as `writerId`: + + - **Shell (`gh`/`kubectl`/`curl`/`git`)** — prefix the fetch with the + wrapper; it behaves exactly like the raw command (same stdout, same exit + code) but only executes on a miss: + `node <pluginRoot>/bin/cached-exec.mjs <buildId> <testRunId> '<command>'` + Wrap ONLY the fetch and pipe *outside* it, so different downstream + filters share one cached fetch: + `node .../cached-exec.mjs "$B" 3895 'gh api repos/o/r/contents/f' | jq -r .content | head -40` + One fetch per call — the wrapper refuses `;`/`&&`/backticks/redirects. + - **MCP data queries** (grafana/VictoriaLogs, `listTestIds`, + `getFailureLogs`) — check first, and store your digest on a miss: + `node <pluginRoot>/bin/cached-mcp.mjs <buildId> get <tool> '<argsJson>'` + (exit 0 = hit, use it and skip the MCP call; exit 1 = miss, make the call + then `... put <tool> '<argsJson>' <testRunId>` with the digest on stdin). + Worth it for expensive build-level queries several coordinators would + each re-run; skip it for a one-off only this test needs, since a miss + costs two extra calls. + - **NEVER cache `tfaRcaTurn` / `getTfaTurnResult` / `triggerRcaReport`** — + they are stateful, and the cache refuses them outright. + - Don't re-probe a connector the gate already validated (`gh auth status`, + `kubectl version`); the manifest above is the answer. 1. **Logs by TFA — the core contract.** Never seed logs in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or digest log content. Logs are TFA's job. diff --git a/bin/cached-exec.mjs b/bin/cached-exec.mjs new file mode 100644 index 0000000..931292d --- /dev/null +++ b/bin/cached-exec.mjs @@ -0,0 +1,93 @@ +#!/usr/bin/env node +// Run a READ-ONLY command through the build's tool cache, in ONE tool call. +// +// Why a wrapper: a "check cache / run / store" sequence done by hand costs +// three tool calls to save one, which is worse than not caching. This collapses +// it to a single call that behaves exactly like the underlying command — +// same stdout, same exit code — but only actually executes on a miss. +// +// Usage (command is ONE argument, so the caller's own quoting survives): +// node bin/cached-exec.mjs <buildId> <writerId> '<command>' +// node bin/cached-exec.mjs <buildId> --stats +// +// Wrap only the expensive fetch and leave filtering to the outer shell: +// node bin/cached-exec.mjs "$B" 3895581484 'gh api repos/o/r/contents/f' | jq -r .content | head -40 +// Two coordinators piping the same fetch through different greps then share +// one cache entry, instead of each paying for the fetch. +// +// Cache hits/misses are reported on STDERR so stdout stays byte-identical to +// the raw command — piping into `grep`/`head`/`jq` is unaffected. + +import { execFileSync } from "node:child_process"; +import { + toolCacheDirFor, cacheKey, cacheGet, cachePut, cacheStats, isRunnable, tokenize, +} from "../lib/tool-cache.mjs"; + +const [, , buildId, writerOrFlag, command] = process.argv; + +if (!buildId || (writerOrFlag !== "--stats" && !command)) { + console.error("usage: cached-exec.mjs <buildId> <writerId> '<command>'"); + console.error(" cached-exec.mjs <buildId> --stats"); + process.exit(2); +} + +const dir = toolCacheDirFor(buildId, process.env.RCA_STATE_DIR ?? ""); + +if (writerOrFlag === "--stats") { + const s = cacheStats(dir); + console.log(JSON.stringify({ cacheDir: dir, ...s }, null, 2)); + process.exit(0); +} + +const key = cacheKey(command); +const hit = cacheGet(dir, key); + +if (hit) { + console.error(`[tool-cache HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + process.stdout.write(hit.stdout); + process.exit(0); +} + +// Gate before running anything (allowlisted read-only leader, no chaining). +const gate = isRunnable(command); +if (!gate.ok) { + console.error(`[tool-cache REFUSED] ${gate.reason}`); + console.error(` command: ${command}`); + process.exit(2); +} + +// No shell: tokenize ourselves and execFile the binary directly, so shell +// metacharacters inside arguments (a --jq expression, an XPath, a LogsQL +// filter) are passed through literally and cannot start a second command. +let argv; +try { + argv = tokenize(command); +} catch (err) { + console.error(`[tool-cache REFUSED] ${err.message}`); + process.exit(2); +} + +let stdout = ""; +let exitCode = 0; +try { + stdout = execFileSync(argv[0], argv.slice(1), { + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + }); +} catch (err) { + // Preserve the real behaviour of the wrapped command: emit whatever it + // produced and exit non-zero. Deliberately NOT cached — a transient + // failure (rate limit, expired token) must not become a permanent answer. + stdout = (err.stdout ?? "").toString(); + exitCode = typeof err.status === "number" ? err.status : 1; + if (err.stderr) process.stderr.write(err.stderr.toString()); + console.error(`[tool-cache MISS ${key} — command exited ${exitCode}, NOT cached]`); + process.stdout.write(stdout); + process.exit(exitCode); +} + +// nowMs is read here, at the process edge, rather than inside lib/ — the +// library keeps its no-clock discipline so it stays sandbox-safe. +cachePut(dir, key, { command, writerId: writerOrFlag, stdout, exitCode }, Date.now()); +console.error(`[tool-cache MISS ${key} — stored ${stdout.length}B]`); +process.stdout.write(stdout); diff --git a/bin/cached-mcp.mjs b/bin/cached-mcp.mjs new file mode 100644 index 0000000..8a57a8e --- /dev/null +++ b/bin/cached-mcp.mjs @@ -0,0 +1,98 @@ +#!/usr/bin/env node +// Memo cache for READ-ONLY **MCP** tool calls, sharing the same per-build +// store as `cached-exec.mjs`. +// +// Shell calls can be wrapped transparently (`cached-exec.mjs` runs the command +// for you). MCP calls cannot — only the agent can invoke an MCP tool — so the +// contract here is check-then-call: +// +// 1. get → node bin/cached-mcp.mjs <buildId> get <tool> '<argsJson>' +// exit 0 + result on stdout = HIT, skip the MCP call entirely +// exit 1, empty stdout = MISS, make the MCP call yourself +// 2. put → node bin/cached-mcp.mjs <buildId> put <tool> '<argsJson>' <writerId> +// (payload on STDIN — pipe the digest you want shared) +// +// WHEN THIS PAYS OFF, and when it does not. A hit replaces one MCP call with +// one cheap local read, so it wins on latency and on tokens whenever the +// cached payload is a digest smaller than the raw response. A miss costs two +// extra calls (the probe + the store), so this is worth it for **expensive, +// broadly-reusable, build-level queries** — a VictoriaLogs sweep, a +// `listTestIds`, a `getFailureLogs` several coordinators would each re-run — +// and NOT worth it for a one-off lookup only this test will ever need. +// +// Never cacheable (refused): `tfaRcaTurn`, `getTfaTurnResult`, +// `triggerRcaReport`. Those are stateful — a turn's status is *expected* to +// change between reads, so serving one from cache is wrong, not just stale. +// Prefer storing a DIGEST rather than a raw payload: the point is to spare the +// next reader the raw rows, not to relay them. + +import { readFileSync } from "node:fs"; +import { + toolCacheDirFor, mcpCacheKey, cacheGet, cachePut, cacheStats, isCacheableMcp, +} from "../lib/tool-cache.mjs"; + +const [, , buildId, verb, tool, argsJson, writerId] = process.argv; + +if (!buildId || !verb) { + console.error("usage: cached-mcp.mjs <buildId> get <tool> '<argsJson>'"); + console.error(" cached-mcp.mjs <buildId> put <tool> '<argsJson>' <writerId> # payload on stdin"); + console.error(" cached-mcp.mjs <buildId> stats"); + process.exit(2); +} + +const dir = toolCacheDirFor(buildId, process.env.RCA_STATE_DIR ?? ""); + +if (verb === "stats") { + console.log(JSON.stringify({ cacheDir: dir, ...cacheStats(dir) }, null, 2)); + process.exit(0); +} + +if (!tool || argsJson === undefined) { + console.error("both <tool> and '<argsJson>' are required"); + process.exit(2); +} + +if (!isCacheableMcp(tool)) { + console.error(`[mcp-cache REFUSED] ${tool} is stateful — never cache it; call it directly.`); + process.exit(2); +} + +let args; +try { + args = JSON.parse(argsJson); +} catch (err) { + console.error(`[mcp-cache] argsJson is not valid JSON: ${err.message}`); + process.exit(2); +} + +const key = mcpCacheKey(tool, args); + +if (verb === "get") { + const hit = cacheGet(dir, key); + if (!hit) { + console.error(`[mcp-cache MISS ${key} ${tool}] — make the MCP call, then 'put' the digest`); + process.exit(1); + } + console.error(`[mcp-cache HIT ${key} ${tool} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + process.stdout.write(hit.stdout); + process.exit(0); +} + +if (verb === "put") { + let payload = ""; + try { + payload = readFileSync(0, "utf8"); // stdin + } catch { + payload = ""; + } + if (!payload.trim()) { + console.error("[mcp-cache] refusing to store an empty payload"); + process.exit(2); + } + const rec = cachePut(dir, key, { command: `${tool} ${argsJson}`, writerId, stdout: payload }, Date.now()); + console.error(`[mcp-cache STORED ${key} ${tool} — ${rec.bytes}B]`); + process.exit(0); +} + +console.error(`unknown verb: ${verb}`); +process.exit(2); diff --git a/config/rca.config.json b/config/rca.config.json index b316f94..aea7ace 100644 --- a/config/rca.config.json +++ b/config/rca.config.json @@ -7,10 +7,12 @@ "turnMessageMaxChars": 1000, "pollSoftPendingMs": 90000, "$softPendingDrainComment": "tfaRcaTurn abandons its in-call poll at pollSoftPendingMs (90s) and returns a soft PENDING while the TFA agent keeps working — turns finalizing past 90s are routine. On a soft PENDING the loop READS the same turnId via getTfaTurnResult on this budget before it routes asks or submits anything further; reads do not consume turnCap. Only when the budget is spent does the run end PENDING (pending-resume row).", + "$maxErrorReadsComment": "A soft PENDING is drained on the full budget below, but a HARD read failure (a thrown error, or a result whose status/message says the TFA agent run failed) is a different signal: it will not resolve by asking again. After this many CONSECUTIVE failed reads the drain stops early and the row ends PENDING with a `tfa-error` note, still resumable. A single good read clears the streak. Measured motivation: on one real build, drain reads plus their sleeps were 23% of all coordinator tool calls, and the four tests that wedged this way were the four slowest in the batch.", "softPendingDrain": { "maxWaitMs": 600000, "intervalMs": 5000, - "maxReads": 40 + "maxReads": 40, + "maxErrorReads": 3 }, "reaperHeartbeatTtlSec": 600, "errorSummaryMaxChars": 200, diff --git a/lib/loop.mjs b/lib/loop.mjs index d4334f4..d9535fa 100644 --- a/lib/loop.mjs +++ b/lib/loop.mjs @@ -32,7 +32,7 @@ import { routeAsks } from "./routing.mjs"; // Drain budget for one soft-PENDING. Bounded so a wedged turn can never hang the // batch — on exhaustion the loop still ends `PENDING` (resumable via the CSV's // `pending-resume` row), which is the old behaviour as a floor, not a default. -const DEFAULT_DRAIN = { maxWaitMs: 600_000, intervalMs: 5_000, maxReads: 40 }; +const DEFAULT_DRAIN = { maxWaitMs: 600_000, intervalMs: 5_000, maxReads: 40, maxErrorReads: 3 }; const defaultSleep = (ms) => new Promise((r) => setTimeout(r, ms)); @@ -40,6 +40,30 @@ const defaultSleep = (ms) => new Promise((r) => setTimeout(r, ms)); const isAgentStatus = (s) => s === "RESOLVED" || s === "NEEDS_INFO" || s === "BLOCKED"; +/** + * Distinguish "TFA is still thinking" from "this read HARD-FAILED". + * + * These deserve opposite responses and the original drain conflated them: + * a `PENDING` should be waited out on the full budget, but a server-side + * `TFA agent run failed` will keep failing, and patiently re-reading it burns + * the entire 40-read / 10-minute budget to learn nothing. Measured on one real + * build: drain reads + their sleeps were 23% of ALL coordinator tool calls, + * and the four agents that wedged this way were the four slowest in the batch. + * + * Only explicit failure signals count — an unrecognised-but-parseable turn is + * treated as "still working", so a new upstream status can never be + * misclassified as an error and cut the drain short. + */ +function isErrorRead(turn, threw) { + if (threw) return true; + if (turn == null) return true; + if (typeof turn === "string") return /\b(fail(ed|ure)?|error)\b/i.test(turn); + if (turn.error) return true; + if (typeof turn.status === "string" && /^(ERROR|FAILED)$/i.test(turn.status)) return true; + if (typeof turn.message === "string" && /\b(fail(ed|ure)?|error)\b/i.test(turn.message)) return true; + return false; +} + function unavailableBlock(gap) { const what = gap?.ask?.what ?? ""; return [ @@ -56,32 +80,47 @@ function unavailableBlock(gap) { // not a new turn. `readTurn` is read-only and side-effect free, so the only // budget that applies is wall clock / read count. // -// Returns { turn, reads }: `turn` is the landed agent turn, or null if the drain -// budget ran out (caller then ends PENDING, resumable). +// Returns { turn, reads, reason }: `turn` is the landed agent turn, or null if +// the drain gave up — `reason` is `landed` | `tfa-error` | `budget-spent` | +// `not-drainable`, which the caller surfaces in the RCA_OUTPUT note so a human +// can tell "TFA was slow" apart from "TFA broke". async function drainSoftPending({ testRunId, pending, readTurn, sleep, drain }) { - const { maxWaitMs, intervalMs, maxReads } = { ...DEFAULT_DRAIN, ...(drain ?? {}) }; + const { maxWaitMs, intervalMs, maxReads, maxErrorReads } = { ...DEFAULT_DRAIN, ...(drain ?? {}) }; const turnId = pending.turnId; let reads = 0; // No turnId → nothing addressable to read; no readTurn → client lacks the // getTfaTurnResult tool. Either way fall back to reporting it resumable. - if (!turnId || typeof readTurn !== "function") return { turn: null, reads }; + if (!turnId || typeof readTurn !== "function") return { turn: null, reads, reason: "not-drainable" }; const started = Date.now(); + let consecutiveErrors = 0; while (reads < maxReads && Date.now() - started < maxWaitMs) { await sleep(intervalMs); reads++; let turn; + let threw = false; try { turn = await readTurn({ testRunId, turnId }); } catch { - // A failed read is not a verdict — keep reading until the budget is spent. + threw = true; + } + + if (isErrorRead(turn, threw)) { + // Hard failure. Allow a couple of retries for a genuine blip, then stop: + // a wedged turn will not un-wedge by being asked the same question 37 + // more times, and the row stays resumable either way. + if (++consecutiveErrors >= maxErrorReads) { + return { turn: null, reads, reason: "tfa-error" }; + } continue; } - if (isAgentStatus(turn?.status)) return { turn, reads }; + + consecutiveErrors = 0; // a good read clears the streak + if (isAgentStatus(turn?.status)) return { turn, reads, reason: "landed" }; // still PENDING → the agent is working; read again. } - return { turn: null, reads }; + return { turn: null, reads, reason: "budget-spent" }; } // runRcaLoop drives one test to a terminal RCA_OUTPUT object. @@ -157,11 +196,13 @@ export async function runRcaLoop({ drain, }); if (!drained.turn) { - return out( - "PENDING", - turn, - `soft-pending: still working after ${drained.reads} read(s)`, - ); + const note = + drained.reason === "tfa-error" + ? `tfa-error: read failed ${drained.reads} time(s) — stopped early, row stays resumable` + : drained.reason === "not-drainable" + ? `soft-pending: no turnId or no getTfaTurnResult tool` + : `soft-pending: still working after ${drained.reads} read(s)`; + return out("PENDING", turn, note); } turn = drained.turn; threadId = turn.threadId ?? threadId; diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs new file mode 100644 index 0000000..94526ef --- /dev/null +++ b/lib/tool-cache.mjs @@ -0,0 +1,222 @@ +// Build-scoped memo cache for READ-ONLY tool calls (`gh`, `kubectl`, curl, …). +// +// The evidence file (`evidence-file.mjs`) shares *digested findings* whose +// shape the schema knows about — deploy state, PRs, log sweeps. But a +// coordinator's real tool traffic is mostly raw lookups that fit no schema +// slot: fetching a spec file's contents at a ref, listing a git tree, running +// a code search. Measured on one real 10-test build: `gh` was 37% of all +// coordinator tool calls, and 46 of them were byte-identical commands re-run +// by different coordinators — the same BStackAutomation spec fetched 12 +// times, a frontend component 5 times. +// +// This module memoizes at the CALL level instead, so duplication is caught +// regardless of what the call was for. Any coordinator about to run a +// read-only command checks here first; on a miss it runs the command and +// stores the result for everyone else. +// +// CONCURRENCY: one file per cache KEY (`<sha>.json`), not one per writer. +// Distinct calls write distinct files; two agents racing on the *same* call +// write byte-identical content, so the race is benign. Writes go through a +// temp file + `rename`, which is atomic on POSIX, so a reader never observes +// a half-written entry. No locking, no lost updates, no torn reads. +// +// WHAT IS NOT CACHED (deliberate): +// - Any command that fails (non-zero exit). A transient `gh` rate-limit or +// an expired kube token must never be memoized into a persistent "answer" +// that poisons every later reader. +// - Anything matching the mutation denylist below. The plugin is read-only +// by contract, but caching is a correctness-sensitive place to trust that +// contract blindly, so mutations are refused defensively. +// - Secrets: values that look like tokens/passwords are redacted from the +// stored payload before it ever touches disk. + +import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, renameSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { createHash } from "node:crypto"; + +/** Per-build cache directory, sitting alongside the state CSV and evidence + * file under the same OS-temp convention. */ +export function toolCacheDirFor(buildId, stateDir = "") { + const safe = String(buildId ?? "").replace(/[^A-Za-z0-9._-]/g, "_") || "unknown-build"; + const dir = stateDir && String(stateDir).trim() !== "" ? String(stateDir) : join(tmpdir(), "bstack-rca"); + return join(dir, `rca-toolcache.${safe}`); +} + +/** Stable key for one call. Whitespace is normalized so trivially-different + * formatting of the same command still hits, but nothing else is rewritten — + * `| head -20` vs `| head -200` genuinely return different output and must + * stay distinct keys. */ +export function cacheKey(command) { + const norm = String(command ?? "").replace(/\s+/g, " ").trim(); + return createHash("sha256").update(norm).digest("hex").slice(0, 24); +} + +// Commands that must never be memoized, even if someone wires this into a +// non-read-only context by mistake. +const MUTATING = /\b(rm|mv|cp|dd|truncate|tee)\b|>\s*\/|\bgit\s+(push|commit|merge|rebase|reset|checkout|clean)\b|\bgh\s+(pr\s+(create|merge|close|edit|comment|review)|issue\s+(create|close|edit|comment)|release\s+create|repo\s+(create|delete)|api\s+(-X\s*)?(POST|PUT|PATCH|DELETE))|\bkubectl\s+(apply|delete|edit|patch|scale|create|replace|annotate|label|cordon|drain|exec|cp|port-forward|rollout\s+(undo|restart|pause|resume))\b|\bcurl\b[^|]*\s-(X|-request)\s*(POST|PUT|PATCH|DELETE)/i; + +export function isCacheable(command) { + return !MUTATING.test(String(command ?? "")); +} + +// Read-only data-fetching binaries this wrapper will run. Anything else is +// refused outright. +const ALLOWED_LEADER = /^\s*(gh|kubectl|curl|git)\s/; + +// Shell constructs that chain a *second* command onto the one being cached: +// `;`, `&&`, `||`, background `&`, command substitution, and output redirects. +// Plain pipes are allowed on purpose — `| jq`, `| grep`, `| head` are how +// callers narrow a fetch, and they don't introduce a new root command. +const CHAINING = /[;&]|\|\||\$\(|`|>>?/; + +/** + * Gate for `bin/cached-exec.mjs`. Returns `{ ok, reason }`. + * + * Scope note, stated plainly: this is defense-in-depth, NOT a security + * boundary. The only caller is a coordinator agent that already has direct + * shell access via its Bash tool, so the wrapper grants no capability the + * caller lacks and cannot meaningfully contain a caller that wants to misuse + * it. What it does buy: a fat-fingered or model-hallucinated command can't + * quietly run something mutating *through the cache path* and get memoized, + * and refusing `;`-chained loops nudges callers toward one-fetch-per-call, + * which caches far better anyway. + */ +export function isRunnable(command) { + const c = String(command ?? ""); + if (!ALLOWED_LEADER.test(c)) { + return { ok: false, reason: "command must start with gh, kubectl, curl, or git" }; + } + if (CHAINING.test(c)) { + return { + ok: false, + reason: "chaining/substitution/redirect not allowed — issue one fetch per call (pipes to jq/grep/head are fine outside the wrapper)", + }; + } + if (!isCacheable(c)) return { ok: false, reason: "command looks mutating" }; + return { ok: true }; +} + +/** + * Split a command string into argv the way a shell would for the simple cases + * we allow — honouring single/double quotes — WITHOUT invoking a shell. The + * caller then runs `execFile(argv[0], argv.slice(1))`, so no shell ever + * interprets metacharacters and command injection has no surface. Throws on + * an unterminated quote rather than guessing. + */ +export function tokenize(command) { + const out = []; + let cur = ""; + let quote = null; + let started = false; + for (const ch of String(command ?? "")) { + if (quote) { + if (ch === quote) quote = null; + else cur += ch; + continue; + } + if (ch === "'" || ch === '"') { quote = ch; started = true; continue; } + if (/\s/.test(ch)) { + if (started) { out.push(cur); cur = ""; started = false; } + continue; + } + cur += ch; + started = true; + } + if (quote) throw new Error("unterminated quote in command"); + if (started) out.push(cur); + return out; +} + +// ---- MCP calls ------------------------------------------------------------ + +// Stateful MCP tools that must NEVER be memoized. `tfaRcaTurn` advances a +// conversation and `getTfaTurnResult` reads a turn whose status is *expected* +// to change between reads — serving either from cache would be actively +// wrong, not merely stale. +const MCP_NEVER = /tfaRcaTurn|getTfaTurnResult|triggerRcaReport/i; + +export function isCacheableMcp(toolName) { + return !MCP_NEVER.test(String(toolName ?? "")); +} + +/** Key an MCP call by tool name + canonicalized args (object keys sorted), so + * the same query written with its arguments in a different order still hits. */ +export function mcpCacheKey(toolName, args) { + const canon = (v) => { + if (Array.isArray(v)) return v.map(canon); + if (v && typeof v === "object") { + return Object.keys(v).sort().reduce((a, k) => { a[k] = canon(v[k]); return a; }, {}); + } + return v; + }; + const payload = JSON.stringify({ tool: String(toolName ?? ""), args: canon(args ?? {}) }); + return createHash("sha256").update(payload).digest("hex").slice(0, 24); +} + +// Redact the REST OF THE LINE after a secret-ish key, not just the next +// token: `Authorization: Bearer <tok>` puts the actual credential in the +// second word, so a `\S+` capture would leave it sitting on disk. +const SECRET_LINE = + /((?:token|authorization|api[_-]?key|secret|password|passwd|bearer|access[_-]?key)\s*[=:]\s*)([^\r\n]*)/gi; + +/** Redact anything token-shaped before it is persisted. The cache lives in + * temp, but a cached `gh api` response or log line could still carry a + * credential, and "it's only temp" is not a reason to write one to disk. */ +export function redact(text) { + return String(text ?? "").replace(SECRET_LINE, (_m, k) => `${k}<redacted>`); +} + +const MAX_BYTES = 256 * 1024; + +export function cacheGet(cacheDir, key) { + const p = join(cacheDir, `${key}.json`); + if (!existsSync(p)) return null; + try { + return JSON.parse(readFileSync(p, "utf8")); + } catch { + return null; // half-written or corrupt -> treat as a miss, never throw + } +} + +/** Atomic write: temp file + rename, so concurrent readers only ever see a + * complete entry. `nowMs` is passed in (same clock discipline as the rest of + * lib/). Returns the stored entry. */ +export function cachePut(cacheDir, key, entry, nowMs) { + if (!existsSync(cacheDir)) mkdirSync(cacheDir, { recursive: true }); + const raw = redact(entry.stdout ?? ""); + const truncated = raw.length > MAX_BYTES; + const rec = { + key, + command: entry.command, + writerId: entry.writerId ?? null, + capturedAtMs: nowMs, + exitCode: entry.exitCode ?? 0, + truncated, + bytes: raw.length, + stdout: truncated ? raw.slice(0, MAX_BYTES) + "\n… [truncated by tool-cache]" : raw, + }; + const finalPath = join(cacheDir, `${key}.json`); + const tmpPath = join(cacheDir, `.${key}.${process.pid}.tmp`); + writeFileSync(tmpPath, JSON.stringify(rec, null, 2), "utf8"); + renameSync(tmpPath, finalPath); // atomic on POSIX + return rec; +} + +/** Cache-wide counters for the run's summary — how much duplicate work this + * actually saved, rather than assuming it saved any. */ +export function cacheStats(cacheDir) { + if (!existsSync(cacheDir)) return { entries: 0, bytes: 0 }; + let entries = 0; + let bytes = 0; + for (const f of readdirSync(cacheDir)) { + if (!f.endsWith(".json") || f.startsWith(".")) continue; + entries++; + try { + bytes += JSON.parse(readFileSync(join(cacheDir, f), "utf8")).bytes ?? 0; + } catch { + /* skip */ + } + } + return { entries, bytes }; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 686441d..a4e6632 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -349,6 +349,17 @@ that agent type — no need to repeat the mechanics in the prompt, just don't omit `evidenceFilePath` (above), since write-back has nothing to write to without it. +**Also hand every dispatch the tool cache.** The evidence file shares digested +*findings*; `bin/cached-exec.mjs` / `bin/cached-mcp.mjs` share raw *call +results*, which is where most duplicate work actually hides — on one measured +build `gh` was 37% of all coordinator tool calls and 46 were byte-identical +commands re-run by different coordinators. Include the plugin root in each +dispatch prompt so coordinators can invoke the wrappers, and tell them to pass +their own `testRunId` as `writerId`. The cache lives at +`<tmpdir>/bstack-rca/rca-toolcache.<buildId>/`, one file per call key, shared +by shell and MCP alike. Read `node bin/cached-exec.mjs <buildId> --stats` at +the end of the run to report how much it actually saved rather than assuming. + **Concurrency is handled by layout, not by locking.** Base (`rca-evidence.<buildId>.json`) has exactly one writer — this orchestrator, in Step 4. Every coordinator writes only its own shard under diff --git a/tests/conformance.test.mjs b/tests/conformance.test.mjs index 33ccbbb..d9af539 100644 --- a/tests/conformance.test.mjs +++ b/tests/conformance.test.mjs @@ -174,6 +174,67 @@ test("a failed read is not a verdict — the drain keeps reading and still lands assert.equal(reads, 2); }); +test("a PERSISTENT hard error stops the drain early instead of burning the budget", async () => { + const fx = load("soft-pending-drain.json"); + let reads = 0; + const result = await runRcaLoop({ + testRunId: fx.testRunId, + submit: replaySubmit([fx.turns[0]]), // always soft-PENDING + readTurn: async () => { + reads++; + throw new Error("Failed to get tfa turn result: TFA agent run failed"); + }, + // Budget allows 40 reads; the error cap must cut it off long before that. + config: { ...CONFIG, softPendingDrain: { maxWaitMs: 600_000, intervalMs: 1, maxReads: 40, maxErrorReads: 3 } }, + sleep: noSleep, + }); + assert.equal(result.status, "PENDING"); + assert.equal(reads, 3, "stopped at maxErrorReads, not the 40-read budget"); + assert.match(result.root_cause, /tfa-error/); + assert.equal(result.turnId, "c2e1a6fd-2243-4f93-bc69-62f298db062c"); // still resumable +}); + +test("an error-shaped RESULT (not thrown) also trips the fast-fail", async () => { + const fx = load("soft-pending-drain.json"); + let reads = 0; + const result = await runRcaLoop({ + testRunId: fx.testRunId, + submit: replaySubmit([fx.turns[0]]), + // The MCP tool reports the wedge as a returned payload, not an exception. + readTurn: async () => { + reads++; + return { status: "ERROR", message: "TFA agent run failed" }; + }, + config: { ...CONFIG, softPendingDrain: { maxWaitMs: 600_000, intervalMs: 1, maxReads: 40, maxErrorReads: 2 } }, + sleep: noSleep, + }); + assert.equal(result.status, "PENDING"); + assert.equal(reads, 2); + assert.match(result.root_cause, /tfa-error/); +}); + +test("INTERMITTENT errors do not trip the fast-fail — a good read clears the streak", async () => { + const fx = load("soft-pending-drain.json"); + const landed = fx.reads[2]; + let reads = 0; + const result = await runRcaLoop({ + testRunId: fx.testRunId, + submit: replaySubmit(fx.turns), + readTurn: async () => { + reads++; + // fail, ok, fail, ok, ... never 2 consecutive failures + if (reads % 2 === 1) throw new Error("transient 502"); + return reads < 6 ? { status: "PENDING" } : landed; + }, + config: { ...CONFIG, softPendingDrain: { maxWaitMs: 600_000, intervalMs: 1, maxReads: 40, maxErrorReads: 2 } }, + manifest: GITHUB_AVAILABLE, + gather, + sleep: noSleep, + }); + assert.equal(result.status, "RESOLVED", "flaky-but-recovering reads must still land"); + assert.equal(reads, 6); +}); + test("BLOCKED surfaced by a drain is terminal — no empty resubmits to the turn cap", async () => { const fx = load("soft-pending-drain.json"); let submits = 0; diff --git a/tests/tool-cache.test.mjs b/tests/tool-cache.test.mjs new file mode 100644 index 0000000..f4549a7 --- /dev/null +++ b/tests/tool-cache.test.mjs @@ -0,0 +1,151 @@ +import { test, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + toolCacheDirFor, cacheKey, mcpCacheKey, cacheGet, cachePut, cacheStats, + isCacheable, isCacheableMcp, isRunnable, redact, tokenize, +} from "../lib/tool-cache.mjs"; + +let dir; +beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "rca-toolcache-")); }); +afterEach(() => rmSync(dir, { recursive: true, force: true })); + +test("toolCacheDirFor: build id in the path, OS temp default, stateDir override", () => { + assert.ok(toolCacheDirFor("b1").startsWith(join(tmpdir(), "bstack-rca"))); + assert.ok(toolCacheDirFor("b1").endsWith("rca-toolcache.b1")); + assert.equal(toolCacheDirFor("b1", "/ci/art"), join("/ci/art", "rca-toolcache.b1")); + assert.ok(toolCacheDirFor("../../etc").endsWith("rca-toolcache..._.._etc")); +}); + +test("cacheKey: whitespace-insensitive, but content-sensitive", () => { + assert.equal(cacheKey("gh api repos/a"), cacheKey("gh api repos/a")); + assert.notEqual(cacheKey("gh api repos/a | head -20"), cacheKey("gh api repos/a | head -200")); +}); + +test("put then get round-trips", () => { + const k = cacheKey("gh api repos/a"); + cachePut(dir, k, { command: "gh api repos/a", writerId: "w1", stdout: "hello" }, 1000); + const hit = cacheGet(dir, k); + assert.equal(hit.stdout, "hello"); + assert.equal(hit.writerId, "w1"); + assert.equal(hit.capturedAtMs, 1000); +}); + +test("get on a miss returns null, never throws", () => { + assert.equal(cacheGet(dir, cacheKey("never run")), null); +}); + +test("a corrupt entry reads as a miss rather than throwing", () => { + const k = cacheKey("gh api repos/a"); + writeFileSync(join(dir, `${k}.json`), "{not json", "utf8"); + assert.equal(cacheGet(dir, k), null); +}); + +test("secrets are redacted before anything is written to disk", () => { + const k = cacheKey("gh api repos/a"); + cachePut(dir, k, { + command: "gh api repos/a", + stdout: 'ok\nAuthorization: Bearer abc123SECRET\napi_key=zzz999\ndone', + }, 1000); + const raw = cacheGet(dir, k).stdout; + assert.ok(!raw.includes("abc123SECRET"), "bearer token must not persist"); + assert.ok(!raw.includes("zzz999"), "api_key must not persist"); + assert.ok(raw.includes("<redacted>")); +}); + +test("redact leaves ordinary output untouched", () => { + assert.equal(redact("just some log output"), "just some log output"); +}); + +test("oversized payloads are truncated and flagged", () => { + const k = cacheKey("gh api big"); + const rec = cachePut(dir, k, { command: "gh api big", stdout: "x".repeat(400 * 1024) }, 1000); + assert.equal(rec.truncated, true); + assert.ok(rec.stdout.includes("[truncated by tool-cache]")); +}); + +test("cacheStats counts entries", () => { + cachePut(dir, "k1", { command: "a", stdout: "12345" }, 1); + cachePut(dir, "k2", { command: "b", stdout: "123" }, 1); + const s = cacheStats(dir); + assert.equal(s.entries, 2); + assert.equal(s.bytes, 8); +}); + +test("isCacheable rejects mutating shell commands", () => { + assert.equal(isCacheable("gh api repos/a"), true); + assert.equal(isCacheable("kubectl get pods"), true); + assert.equal(isCacheable("kubectl delete pod x"), false); + assert.equal(isCacheable("kubectl exec pod -- sh"), false); + assert.equal(isCacheable("gh pr create --title x"), false); + assert.equal(isCacheable("gh api -X POST repos/a"), false); + assert.equal(isCacheable("git push origin main"), false); + assert.equal(isCacheable("rm -rf /tmp/x"), false); +}); + +test("isRunnable enforces an allowlisted read-only leader", () => { + assert.equal(isRunnable("gh api repos/a").ok, true); + assert.equal(isRunnable("kubectl get pods -n regression").ok, true); + assert.equal(isRunnable("python3 -c 'print(1)'").ok, false); + assert.equal(isRunnable("sh -c 'echo hi'").ok, false); +}); + +test("isRunnable rejects chaining, substitution and redirects", () => { + assert.equal(isRunnable("gh api a; rm -rf /").ok, false); + assert.equal(isRunnable("gh api a && kubectl delete pod x").ok, false); + assert.equal(isRunnable("gh api $(whoami)").ok, false); + assert.equal(isRunnable("gh api a > /etc/passwd").ok, false); + assert.equal(isRunnable("gh api a `id`").ok, false); +}); + +test("tokenize splits like a shell for quoted args, without a shell", () => { + assert.deepEqual(tokenize("gh api repos/a --jq '.items[].path'"), + ["gh", "api", "repos/a", "--jq", ".items[].path"]); + assert.deepEqual(tokenize('kubectl get pods -o "custom:.metadata.name"'), + ["kubectl", "get", "pods", "-o", "custom:.metadata.name"]); + assert.throws(() => tokenize("gh api 'unterminated"), /unterminated quote/); +}); + +test("tokenize keeps injection payloads as ONE literal argument", () => { + // With execFile + these argv, no shell ever sees the metacharacters. + const argv = tokenize(`gh api "repos/a;rm -rf /"`); + assert.deepEqual(argv, ["gh", "api", "repos/a;rm -rf /"]); +}); + +test("MCP: stateful tools are never cacheable", () => { + assert.equal(isCacheableMcp("mcp__grafana__query_loki_logs"), true); + assert.equal(isCacheableMcp("mcp__browserstack__listTestIds"), true); + assert.equal(isCacheableMcp("mcp__browserstack__tfaRcaTurn"), false); + assert.equal(isCacheableMcp("mcp__browserstack__getTfaTurnResult"), false); + assert.equal(isCacheableMcp("mcp__browserstack__triggerRcaReport"), false); +}); + +test("mcpCacheKey is argument-order independent but value sensitive", () => { + const a = mcpCacheKey("t", { b: 2, a: 1 }); + const b = mcpCacheKey("t", { a: 1, b: 2 }); + assert.equal(a, b); + assert.notEqual(a, mcpCacheKey("t", { a: 1, b: 3 })); + assert.notEqual(a, mcpCacheKey("other", { a: 1, b: 2 })); +}); + +test("mcpCacheKey canonicalizes nested objects and arrays", () => { + assert.equal( + mcpCacheKey("t", { q: { z: 1, y: [{ n: 1, m: 2 }] } }), + mcpCacheKey("t", { q: { y: [{ m: 2, n: 1 }], z: 1 } }), + ); +}); + +test("an MCP result round-trips through the shared store", () => { + const k = mcpCacheKey("mcp__grafana__query_loki_logs", { ns: "regression", limit: 50 }); + cachePut(dir, k, { command: "grafana query", writerId: "3889074893", stdout: "0 rows, clean" }, 1000); + assert.equal(cacheGet(dir, k).stdout, "0 rows, clean"); +}); + +test("CONCURRENCY: same key written twice stays readable and consistent", () => { + const k = cacheKey("gh api repos/a"); + cachePut(dir, k, { command: "gh api repos/a", writerId: "w1", stdout: "same-bytes" }, 1000); + cachePut(dir, k, { command: "gh api repos/a", writerId: "w2", stdout: "same-bytes" }, 2000); + assert.equal(cacheGet(dir, k).stdout, "same-bytes"); +}); diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 2a7a0e4..4992bee 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -23,7 +23,8 @@ export const meta = { // { // csvPath, buildId, // manifest: { capability: { available, via } }, -// evidenceFilePath, // NEW — lib/evidence-file.mjs artifact for this build +// evidenceFilePath, // lib/evidence-file.mjs artifact for this build +// pluginRoot, // absolute path to this plugin, so coordinators can call bin/cached-exec.mjs // buildEvidence: { baselineRef, isFallback, suspectWindow, reposCovered, workloadsCovered, gaps }, // // ^ SHRUNK to a cheap summary/pointer only — the full PR list / log // // sweeps live in the file at evidenceFilePath, read via each @@ -68,6 +69,7 @@ const shared = [ `Build-evidence summary (full detail is in the file above; this is only a pointer — do not re-fetch what the file already covers): ${JSON.stringify(ctx.buildEvidence ?? {})}`, `If the file's github/logs sections do not name a repo/workload/ask you need, or record a "gap" for it, that is a genuine gap — fall back to a live gather via the capability manifest above exactly as if no file existed. The file is an optimization, never a hard dependency.`, `The file is read-write: after any live gather that fills a gap or goes deeper than what was there, write it back via contributeGithubEvidence/contributeLogsEvidence (lib/evidence-file.mjs) passing your own testRunId as writerId, before finishing this test — so a sibling dispatched after you, or another cluster sharing the same repo/workload, reads the enriched entry instead of re-fetching it. Each writer owns its own shard file, so concurrent coordinators cannot clobber each other; readers fold base + shards automatically.`, + `Tool cache — route read-only lookups through it so duplicate calls across coordinators become hits. Shell: node ${ctx.pluginRoot ?? "<pluginRoot>"}/bin/cached-exec.mjs <buildId> <yourTestRunId> '<gh|kubectl|curl|git command>' (behaves like the raw command; pipe to jq/grep OUTSIDE the wrapper so different filters share one fetch). MCP data queries: cached-mcp.mjs <buildId> get|put <tool> '<argsJson>' [writerId]. NEVER cache tfaRcaTurn/getTfaTurnResult/triggerRcaReport — they are stateful. Do not re-probe connectors the gate already validated.`, `Autonomous run — on an evidence gap with no valid connector, report "unavailable" back to TFA (NEVER prompt a user). Best-effort finalize.`, `PRODUCT_BUG / application-bug mandate: hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure signature) and feed the PR link(s) to TFA so related_prs populates. No PR after digging to the turn cap → state explicitly "no culprit PR identified after <what was searched>" so the CSV row records the gap.`, `Soft-PENDING is NOT an answer: tfaRcaTurn abandons its in-call poll at 90s while TFA keeps working. On status PENDING, call getTfaTurnResult(testRunId, turnId) FIRST and keep reading on the softPendingDrain budget (every 5s, <=40 reads / <=10min) until the status is RESOLVED / NEEDS_INFO / BLOCKED, then continue the loop. Reads do NOT count against the turn cap. Never submit a new message onto a turn still in flight. Only a fully spent drain budget ends the test PENDING.`, From 6cabb9df87eed6634b8a2b4b8547b024bd8f77f3 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 16:45:23 +0530 Subject: [PATCH 11/51] fix(security): write RCA state, evidence and tool cache owner-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three on-disk artifacts live under a world-readable OS temp dir and carry sensitive content: the tool cache holds raw gh/kubectl output (private repo source, internal hostnames, log bodies), the evidence file holds PR detail and app-log digests, and the state CSV holds root causes and culprit PRs. Default umask left them 0644 — readable by any local user. Create directories 0700 and write files 0600. The file mode is the load-bearing control: a pre-existing directory keeps its own permissions (silently chmod'ing a caller-supplied stateDir would be presumptuous), but entries are 0600 either way, and a traversable cache dir only exposes opaque hash filenames. Note this hardens storage, not the redaction: redact() is best-effort pattern matching over token-shaped strings and was never sufficient on its own to make these files safe to leave world-readable. Verified on disk for both a fresh dir (0700/0600) and a pre-existing 0755 dir (files still 0600). 111 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/csv-state.mjs | 6 ++++-- lib/evidence-file.mjs | 10 ++++++---- lib/tool-cache.mjs | 23 +++++++++++++++++++---- tests/evidence-file.test.mjs | 9 ++++++++- tests/tool-cache.test.mjs | 18 +++++++++++++++++- 5 files changed, 54 insertions(+), 12 deletions(-) diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 152e51b..c0723e6 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -140,10 +140,12 @@ export function readRows(csvPath) { }); } +// Owner-only (0700 dir / 0600 file): the state CSV lives in a world-readable +// OS temp dir and records root causes, culprit PRs and evidence digests. export function writeRows(csvPath, rows) { const dir = dirname(csvPath); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true }); - writeFileSync(csvPath, encodeRows(rows), "utf8"); + if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + writeFileSync(csvPath, encodeRows(rows), { encoding: "utf8", mode: 0o600 }); } function emptyRow() { diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index b9010b6..c885309 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -190,10 +190,12 @@ export function readEvidenceFile(filePath) { return doc; } +// Owner-only (0700 dir / 0600 file): this sits in a world-readable OS temp dir +// and carries private-repo PR detail and app-log digests. export function writeEvidenceFile(filePath, doc) { const dir = dirname(filePath); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true }); - writeFileSync(filePath, JSON.stringify(doc, null, 2), "utf8"); + if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + writeFileSync(filePath, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); } function loadOrInit(filePath, nowMs) { @@ -274,8 +276,8 @@ function loadOwnShard(basePath, writerId, nowMs) { function writeShard(path, doc) { const dir = dirname(path); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true }); - writeFileSync(path, JSON.stringify(doc, null, 2), "utf8"); + if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + writeFileSync(path, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); } /** Contribute what THIS coordinator gathered live for a repo — a deeper diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs index 94526ef..4f1507a 100644 --- a/lib/tool-cache.mjs +++ b/lib/tool-cache.mjs @@ -169,6 +169,8 @@ export function redact(text) { const MAX_BYTES = 256 * 1024; +let tmpSeq = 0; + export function cacheGet(cacheDir, key) { const p = join(cacheDir, `${key}.json`); if (!existsSync(p)) return null; @@ -183,7 +185,18 @@ export function cacheGet(cacheDir, key) { * complete entry. `nowMs` is passed in (same clock discipline as the rest of * lib/). Returns the stored entry. */ export function cachePut(cacheDir, key, entry, nowMs) { - if (!existsSync(cacheDir)) mkdirSync(cacheDir, { recursive: true }); + // Owner-only (0700 dir / 0600 files). The cache lives under a world-readable + // OS temp dir and holds raw `gh`/`kubectl` output — private repo source, + // internal hostnames, log bodies. `redact()` below is best-effort pattern + // matching and will not catch everything, so the filesystem permission is + // the actual control, not a backstop. + // + // The FILE mode is the load-bearing part: a pre-existing directory (e.g. a + // `stateDir` the user already created, or one left by an earlier run) keeps + // its own permissions, since silently chmod'ing a path we were handed would + // be presumptuous. Entries stay 0600 regardless, and a traversable directory + // only exposes opaque hash filenames, not their contents. + if (!existsSync(cacheDir)) mkdirSync(cacheDir, { recursive: true, mode: 0o700 }); const raw = redact(entry.stdout ?? ""); const truncated = raw.length > MAX_BYTES; const rec = { @@ -197,9 +210,11 @@ export function cachePut(cacheDir, key, entry, nowMs) { stdout: truncated ? raw.slice(0, MAX_BYTES) + "\n… [truncated by tool-cache]" : raw, }; const finalPath = join(cacheDir, `${key}.json`); - const tmpPath = join(cacheDir, `.${key}.${process.pid}.tmp`); - writeFileSync(tmpPath, JSON.stringify(rec, null, 2), "utf8"); - renameSync(tmpPath, finalPath); // atomic on POSIX + // pid + counter keeps the temp name unique per writer, so `mode` genuinely + // applies (it is honoured on create, not on truncate of an existing file). + const tmpPath = join(cacheDir, `.${key}.${process.pid}.${tmpSeq++}.tmp`); + writeFileSync(tmpPath, JSON.stringify(rec, null, 2), { encoding: "utf8", mode: 0o600 }); + renameSync(tmpPath, finalPath); // atomic on POSIX; preserves the 0600 mode return rec; } diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index dc40742..033d2f5 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -1,6 +1,6 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync, statSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -241,6 +241,13 @@ test("recomputeCoverage counts a coordinator-filled gap as covered", () => { assert.deepEqual(cov.reposGapped, []); }); +test("evidence file and contribution shards are owner-only (0600)", () => { + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "private PR detail" } }, 1000); + contributeGithubEvidence(file, "w1", "org/a", { prsInWindow: [{ pr: "#1" }] }, 2000); + assert.equal(statSync(file).mode & 0o777, 0o600); + assert.equal(statSync(contribPathFor(file, "w1")).mode & 0o777, 0o600); +}); + test("writeEvidenceFile creates the parent directory if missing", () => { const nested = join(dir, "nested", "sub", "evidence.json"); writeEvidenceFile(nested, emptyEvidenceFile("build-1", 0)); diff --git a/tests/tool-cache.test.mjs b/tests/tool-cache.test.mjs index f4549a7..91ea9fb 100644 --- a/tests/tool-cache.test.mjs +++ b/tests/tool-cache.test.mjs @@ -1,6 +1,6 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync, statSync, readdirSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -143,6 +143,22 @@ test("an MCP result round-trips through the shared store", () => { assert.equal(cacheGet(dir, k).stdout, "0 rows, clean"); }); +test("cache files are owner-only (0600) and the dir owner-only (0700)", () => { + const sub = join(dir, "nested-cache"); + const k = cacheKey("gh api repos/a"); + cachePut(sub, k, { command: "gh api repos/a", stdout: "private repo source" }, 1000); + // The cache sits in a world-readable OS temp dir and holds raw gh/kubectl + // output; redaction is best-effort, so the mode is the real control. + assert.equal(statSync(join(sub, `${k}.json`)).mode & 0o777, 0o600); + assert.equal(statSync(sub).mode & 0o777, 0o700); +}); + +test("no temp file is left behind after an atomic put", () => { + const k = cacheKey("gh api repos/a"); + cachePut(dir, k, { command: "gh api repos/a", stdout: "x" }, 1000); + assert.deepEqual(readdirSync(dir).filter((f) => f.endsWith(".tmp")), []); +}); + test("CONCURRENCY: same key written twice stays readable and consistent", () => { const k = cacheKey("gh api repos/a"); cachePut(dir, k, { command: "gh api repos/a", writerId: "w1", stdout: "same-bytes" }, 1000); From 8e4a6d082144494d381454c55d1670de56f4632a Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 18:49:26 +0530 Subject: [PATCH 12/51] fix(rca): six defects found by running the cache against a real build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dispatched three coordinators concurrently on a live build. They surfaced issues no unit test had, worst first: 1. CORRECTNESS: an empty `prsInWindow` with `gap: null` is byte-identical whether the PR search ran and found nothing or was never populated. Seen live — the file asserted 0 PRs for a repo that actually had 21 (the loader's search had silently returned empty), which would let a coordinator report "no culprit PR identified" with false confidence. Add an explicit `prsSearched` flag (sticky across contributors), a `hasTrustworthyPrList` helper, and a `coverage.reposWithUntrustedPrList` signal. Repo-level coverage semantics are unchanged — trustworthiness is reported alongside, not folded into, covered/gapped. 2. The runnable-guard scanned the raw command string, so it refused legitimate read-only calls: `;` inside a jq expression, `&` inside a quoted URL. Check whole argv TOKENS instead — post-tokenization a quoted metacharacter is inside an argument (harmless, we execFile) while a real operator is its own token. Verified both previously-refused commands now run. 3. execFileSync BOTH inherits and captures stderr, so relaying err.stderr printed failures three times. Capture only, relay once — wrapper stderr is now byte-identical to the command's own, plus one banner line. 4. Empty results were cached, making a sticky invisible negative. Not stored. 5. Nested single quotes made `--jq '...'` unpassable inside a quoted command argument. Accept `-` to read the command from stdin. 6. File mode applies on create only, so files left by a pre-hardening run kept 0644 forever. chmod explicitly on overwrite too. Also documented two traps that are usage, not bugs: `2>&1 | jq` merges the stderr banner into the pipe (the banner is correctly on stderr — one agent misreported this as a stdout bug; measured to confirm), and `cached-mcp get`'s exit code is lost behind a pipe. 118 tests pass, including regression tests for each fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 19 ++++++++++++++ bin/cached-exec.mjs | 50 +++++++++++++++++++++++++++++++++--- lib/csv-state.mjs | 5 +++- lib/evidence-file.mjs | 45 +++++++++++++++++++++++++++++++- lib/tool-cache.mjs | 33 +++++++++++++++++------- tests/evidence-file.test.mjs | 45 +++++++++++++++++++++++++++++++- tests/tool-cache.test.mjs | 26 ++++++++++++++++--- 7 files changed, 203 insertions(+), 20 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 1ae0096..163fe0d 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -136,6 +136,25 @@ read-only and has no side effects, so a read is always safe to repeat. they are stateful, and the cache refuses them outright. - Don't re-probe a connector the gate already validated (`gh auth status`, `kubectl version`); the manifest above is the answer. + - Two wrapper gotchas, both hit in real use: **(i)** hit/miss banners go to + stderr so `| jq` works, but `2>&1 | jq` merges the banner into the pipe + and jq dies on it — don't redirect stderr into a pipe. **(ii)** a command + containing its own single quotes (e.g. `--jq '.[] | "\(.number)"'`) can't + be nested inside a single-quoted argument; pipe it in on stdin instead: + `printf '%s' '<command>' | node .../cached-exec.mjs <buildId> <writerId> -`. + Metacharacters *inside* a quoted argument are fine — only a standalone + shell operator is refused, and a pipe belongs outside the wrapper anyway. + + **Never read an empty `prsInWindow` as "no PRs in the window."** An empty + list means "no PRs" ONLY when the entry also has `prsSearched: true`; + otherwise it was never populated and the two are indistinguishable in the + data. Check `coverage.reposWithUntrustedPrList` (or call + `hasTrustworthyPrList(doc, repo)`) before concluding anything from an empty + list — and when it is untrusted, run the PR search live. This is not + hypothetical: a pre-fetch once asserted 0 PRs for a repo that had 21, + which would have produced a confident "no culprit PR identified." When you + do run the search, contribute the result back — that records + `prsSearched` and spares everyone else the same trap. 1. **Logs by TFA — the core contract.** Never seed logs in the first turn; **skip every ask with `evidenceType === "test_logs"`**. Never fetch, paste, or digest log content. Logs are TFA's job. diff --git a/bin/cached-exec.mjs b/bin/cached-exec.mjs index 931292d..d0c6fad 100644 --- a/bin/cached-exec.mjs +++ b/bin/cached-exec.mjs @@ -8,6 +8,7 @@ // // Usage (command is ONE argument, so the caller's own quoting survives): // node bin/cached-exec.mjs <buildId> <writerId> '<command>' +// node bin/cached-exec.mjs <buildId> <writerId> - # command on STDIN // node bin/cached-exec.mjs <buildId> --stats // // Wrap only the expensive fetch and leave filtering to the outer shell: @@ -15,15 +16,43 @@ // Two coordinators piping the same fetch through different greps then share // one cache entry, instead of each paying for the fetch. // -// Cache hits/misses are reported on STDERR so stdout stays byte-identical to -// the raw command — piping into `grep`/`head`/`jq` is unaffected. +// TWO GOTCHAS, both hit in real use: +// +// 1. Hit/miss banners go to STDERR, so stdout stays byte-identical to the raw +// command and `| jq` works. But `2>&1 | jq` merges the banner back into +// the pipe and jq dies on it ("Invalid literal at line 1, column 12"). +// Don't redirect stderr into a pipe; if you must silence it, `2>/dev/null` +// — though that also hides whether you got a hit. +// +// 2. Nested single quotes. A command containing its own `'…'` (typically +// `--jq '.[] | "\(.number)"'`) cannot be passed inside a single-quoted +// argument — the outer shell terminates the string early and the argument +// arrives mangled. Use `-` and pipe the command in on stdin instead: +// printf '%s' 'gh pr list -R o/r --json number --jq ".[].number"' \ +// | node bin/cached-exec.mjs "$B" 3895 - import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; import { toolCacheDirFor, cacheKey, cacheGet, cachePut, cacheStats, isRunnable, tokenize, } from "../lib/tool-cache.mjs"; -const [, , buildId, writerOrFlag, command] = process.argv; +const [, , buildId, writerOrFlag, commandArg] = process.argv; + +// `-` means the command arrives on stdin, which sidesteps the nested-quoting +// problem entirely (see gotcha 2 above). +let command = commandArg; +if (command === "-") { + try { + command = readFileSync(0, "utf8").trim(); + } catch { + command = ""; + } + if (!command) { + console.error("[tool-cache] '-' given but stdin was empty"); + process.exit(2); + } +} if (!buildId || (writerOrFlag !== "--stats" && !command)) { console.error("usage: cached-exec.mjs <buildId> <writerId> '<command>'"); @@ -73,6 +102,10 @@ try { stdout = execFileSync(argv[0], argv.slice(1), { encoding: "utf8", maxBuffer: 64 * 1024 * 1024, + // Capture stderr instead of letting it inherit. Default execFileSync + // BOTH inherits stderr to the parent AND captures it on the error, so + // relaying `err.stderr` ourselves printed everything three times. + stdio: ["ignore", "pipe", "pipe"], }); } catch (err) { // Preserve the real behaviour of the wrapped command: emit whatever it @@ -80,12 +113,21 @@ try { // failure (rate limit, expired token) must not become a permanent answer. stdout = (err.stdout ?? "").toString(); exitCode = typeof err.status === "number" ? err.status : 1; - if (err.stderr) process.stderr.write(err.stderr.toString()); + if (err.stderr) process.stderr.write(err.stderr.toString()); // now the only copy console.error(`[tool-cache MISS ${key} — command exited ${exitCode}, NOT cached]`); process.stdout.write(stdout); process.exit(exitCode); } +// An empty result is not stored. It is usually a wrong selector or a silently +// failed lookup, and caching it makes a sticky, invisible negative that every +// later reader inherits — the expensive kind of wrong. +if (stdout.trim() === "") { + console.error(`[tool-cache MISS ${key} — empty result, NOT cached]`); + process.stdout.write(stdout); + process.exit(0); +} + // nowMs is read here, at the process edge, rather than inside lib/ — the // library keeps its no-clock discipline so it stays sandbox-safe. cachePut(dir, key, { command, writerId: writerOrFlag, stdout, exitCode }, Date.now()); diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index c0723e6..724eea3 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -13,7 +13,7 @@ // is sufficient for the in-process 5-concurrent workflow (true multi-process // locking is out of scope). -import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs"; +import { readFileSync, writeFileSync, existsSync, mkdirSync, chmodSync } from "node:fs"; import { dirname, join } from "node:path"; import { tmpdir } from "node:os"; @@ -145,7 +145,10 @@ export function readRows(csvPath) { export function writeRows(csvPath, rows) { const dir = dirname(csvPath); if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + const existed = existsSync(csvPath); writeFileSync(csvPath, encodeRows(rows), { encoding: "utf8", mode: 0o600 }); + // `mode` applies on create only — tighten a pre-hardening leftover too. + if (existed) chmodSync(csvPath, 0o600); } function emptyRow() { diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index c885309..2e76215 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -58,7 +58,7 @@ // 3. `prsInWindow` is unioned by PR number rather than replaced, so two // coordinators finding different PRs in the same repo both survive. -import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync } from "node:fs"; +import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, chmodSync } from "node:fs"; import { dirname, join } from "node:path"; import { tmpdir } from "node:os"; @@ -143,6 +143,10 @@ function foldGithub(target, repo, entry) { const next = { deployState: pickLeaf(cur.deployState, entry.deployState), prsInWindow: cur.prsInWindow ?? [], + // Sticky: once ANY writer has genuinely run the PR search, the entry stays + // trustworthy — a later contributor that didn't search must not silently + // downgrade it back to "unknown". + prsSearched: cur.prsSearched === true || entry.prsSearched === true, gap: cur.gap ?? null, }; if (Array.isArray(entry.prsInWindow)) { @@ -195,7 +199,12 @@ export function readEvidenceFile(filePath) { export function writeEvidenceFile(filePath, doc) { const dir = dirname(filePath); if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + const existed = existsSync(filePath); writeFileSync(filePath, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); + // `mode` is only honoured when the file is CREATED. A file left over from a + // run that predates this hardening would otherwise keep its old 0644 + // forever, so tighten it explicitly on overwrite too. + if (existed) chmodSync(filePath, 0o600); } function loadOrInit(filePath, nowMs) { @@ -294,7 +303,11 @@ export function contributeGithubEvidence(basePath, writerId, repo, patch, nowMs) const byPr = new Map((entry.prsInWindow ?? []).map((p) => [String(p.pr), p])); for (const pr of patch.prsInWindow) byPr.set(String(pr.pr), pr); entry.prsInWindow = [...byPr.values()]; + // Contributing a list — even an empty one — means you actually ran the + // search, so record that. Pass `prsSearched: false` explicitly to opt out. + entry.prsSearched = patch.prsSearched !== false; } + if (patch.prsSearched !== undefined) entry.prsSearched = patch.prsSearched; if (patch.gap !== undefined) entry.gap = patch.gap; doc.github[repo] = entry; doc.generatedAtMs = nowMs; @@ -324,11 +337,34 @@ export function contributeLogsEvidence(basePath, writerId, workload, patch, nowM // is falsy. Presence with a `gap` is a recorded, deliberate miss — not // coverage — so a coordinator (or this function) never mistakes "we looked // and couldn't get it" for "we have it." +// +// For a github entry there is a further trap, hit for real in testing: an +// empty `prsInWindow` is byte-identical whether the PR search RAN and found +// nothing, or was never populated at all. A coordinator trusting the former +// reads "no PRs in window" and concludes "no culprit PR" — confidently wrong. +// (Observed: a file asserting 0 PRs for a repo that actually had 21, because +// the loader's search silently returned empty.) So an empty `prsInWindow` +// only counts as coverage when `prsSearched === true` explicitly records that +// the search was really performed. +// Kept deliberately at the REPO level: an entry with deploy state but no PR +// search is still real coverage of that repo. PR-list trustworthiness is a +// narrower question, reported separately as `reposWithUntrustedPrList` so it +// is visible without distorting covered/gapped. function isCovered(doc, section, key) { const entry = doc[section]?.[key]; return Boolean(entry) && !entry.gap; } +/** True when this repo entry can be trusted to answer "which PRs were in the + * window" — i.e. it either lists PRs, or explicitly records that the search + * ran and legitimately found none. Coordinators should call this before + * concluding "no culprit PR" from the pre-fetch. */ +export function hasTrustworthyPrList(doc, repo) { + const entry = doc?.github?.[repo]; + if (!entry || entry.gap) return false; + return (entry.prsInWindow ?? []).length > 0 || entry.prsSearched === true; +} + /** Derives `doc.coverage` from exactly which requested repos/workloads have a * gap-free entry, and persists it. `requested = {repos:[...], workloads:[...]}` * — normally the Gate Part A scope-probe-validated repo list and the union of @@ -346,6 +382,13 @@ export function recomputeCoverage(filePath, requested, nowMs) { reposGapped: repos.filter((r) => !isCovered(folded, "github", r)), workloadsCovered: workloads.filter((w) => isCovered(folded, "logs", w)), workloadsGapped: workloads.filter((w) => !isCovered(folded, "logs", w)), + // Repos whose PR list must NOT be read as "no PRs in window": the list is + // empty and nothing recorded that a search actually ran. Surfacing this + // separately keeps a coordinator from concluding "no culprit PR" off an + // array that was simply never filled in. + reposWithUntrustedPrList: repos.filter( + (r) => isCovered(folded, "github", r) && !hasTrustworthyPrList(folded, r), + ), }; doc.generatedAtMs = nowMs; writeEvidenceFile(filePath, doc); diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs index 4f1507a..01cf43c 100644 --- a/lib/tool-cache.mjs +++ b/lib/tool-cache.mjs @@ -30,7 +30,9 @@ // - Secrets: values that look like tokens/passwords are redacted from the // stored payload before it ever touches disk. -import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, renameSync } from "node:fs"; +import { + readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, renameSync, chmodSync, +} from "node:fs"; import { join } from "node:path"; import { tmpdir } from "node:os"; import { createHash } from "node:crypto"; @@ -64,11 +66,15 @@ export function isCacheable(command) { // refused outright. const ALLOWED_LEADER = /^\s*(gh|kubectl|curl|git)\s/; -// Shell constructs that chain a *second* command onto the one being cached: -// `;`, `&&`, `||`, background `&`, command substitution, and output redirects. -// Plain pipes are allowed on purpose — `| jq`, `| grep`, `| head` are how -// callers narrow a fetch, and they don't introduce a new root command. -const CHAINING = /[;&]|\|\||\$\(|`|>>?/; +// Shell operators, checked as whole ARGV TOKENS rather than by scanning the +// raw string. Scanning the string was too blunt and refused legitimate reads: +// `--jq 'test("rcaThree";"i")'` was rejected for the `;` inside a quoted jq +// expression, and `search/code?q=X&per_page=20` for the `&` inside a URL. +// +// Post-tokenization this distinction is exact: quoted metacharacters end up +// *inside* an argument (harmless — we execFile, so no shell ever interprets +// them), while a real operator survives as its own standalone token. +const OPERATOR_TOKENS = new Set([";", "|", "||", "&&", "&", ">", ">>", "<", "<<"]); /** * Gate for `bin/cached-exec.mjs`. Returns `{ ok, reason }`. @@ -87,13 +93,22 @@ export function isRunnable(command) { if (!ALLOWED_LEADER.test(c)) { return { ok: false, reason: "command must start with gh, kubectl, curl, or git" }; } - if (CHAINING.test(c)) { + if (!isCacheable(c)) return { ok: false, reason: "command looks mutating" }; + + let argv; + try { + argv = tokenize(c); + } catch (err) { + return { ok: false, reason: err.message }; + } + const op = argv.find((t) => OPERATOR_TOKENS.has(t)); + if (op) { return { ok: false, - reason: "chaining/substitution/redirect not allowed — issue one fetch per call (pipes to jq/grep/head are fine outside the wrapper)", + reason: `'${op}' is a shell operator — run it OUTSIDE the wrapper (e.g. \`cached-exec … 'gh api X' | jq .y\`) so one cached fetch can serve several different filters. Metacharacters inside a quoted argument are fine.`, }; } - if (!isCacheable(c)) return { ok: false, reason: "command looks mutating" }; + if (argv.length === 0) return { ok: false, reason: "empty command" }; return { ok: true }; } diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index 033d2f5..e00cc2c 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -1,6 +1,6 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, writeFileSync, statSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync, statSync, chmodSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -17,6 +17,7 @@ import { contribDirFor, contribPathFor, readBaseFile, + hasTrustworthyPrList, recomputeCoverage, } from "../lib/evidence-file.mjs"; @@ -241,6 +242,48 @@ test("recomputeCoverage counts a coordinator-filled gap as covered", () => { assert.deepEqual(cov.reposGapped, []); }); +// Regression: an empty prsInWindow with gap:null used to read as "searched, +// found none" when it may simply never have been populated. Observed live — +// a file asserted 0 PRs for a repo that actually had 21, which would have let +// a coordinator conclude "no culprit PR" with false confidence. +test("empty prsInWindow is NOT coverage unless the search is recorded", () => { + setGithubEvidence(file, "org/never-searched", { gap: null, deployState: { block: "d" }, prsInWindow: [] }, 1000); + setGithubEvidence(file, "org/searched-empty", { gap: null, deployState: { block: "d" }, prsInWindow: [], prsSearched: true }, 1000); + const cov = recomputeCoverage(file, { repos: ["org/never-searched", "org/searched-empty"], workloads: [] }, 2000); + // Both repos ARE covered (each has deploy state) — but only one has a PR + // list safe to read as "no PRs in window". + assert.deepEqual(cov.reposCovered.sort(), ["org/never-searched", "org/searched-empty"]); + assert.deepEqual(cov.reposWithUntrustedPrList, ["org/never-searched"]); +}); + +test("hasTrustworthyPrList distinguishes searched-empty from never-populated", () => { + setGithubEvidence(file, "org/a", { gap: null, prsInWindow: [] }, 1000); + setGithubEvidence(file, "org/b", { gap: null, prsInWindow: [], prsSearched: true }, 1000); + setGithubEvidence(file, "org/c", { gap: null, prsInWindow: [{ pr: "#1" }] }, 1000); + const doc = readEvidenceFile(file); + assert.equal(hasTrustworthyPrList(doc, "org/a"), false); + assert.equal(hasTrustworthyPrList(doc, "org/b"), true); + assert.equal(hasTrustworthyPrList(doc, "org/c"), true); +}); + +test("contributing a PR list records that the search actually ran", () => { + contributeGithubEvidence(file, "w1", "org/a", { prsInWindow: [] }, 1000); + assert.equal(hasTrustworthyPrList(readEvidenceFile(file), "org/a"), true); +}); + +test("prsSearched is sticky — a later non-searching contributor cannot downgrade it", () => { + setGithubEvidence(file, "org/a", { gap: null, prsInWindow: [{ pr: "#1" }], prsSearched: true }, 1000); + contributeGithubEvidence(file, "w1", "org/a", { deployState: { block: "just deploy info" } }, 2000); + assert.equal(readEvidenceFile(file).github["org/a"].prsSearched, true); +}); + +test("a pre-existing loose-mode file is tightened to 0600 on the next write", () => { + writeEvidenceFile(file, emptyEvidenceFile("b", 0)); + chmodSync(file, 0o644); // simulate a file left by a pre-hardening run + setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "x" } }, 1000); + assert.equal(statSync(file).mode & 0o777, 0o600); +}); + test("evidence file and contribution shards are owner-only (0600)", () => { setGithubEvidence(file, "org/a", { gap: null, deployState: { block: "private PR detail" } }, 1000); contributeGithubEvidence(file, "w1", "org/a", { prsInWindow: [{ pr: "#1" }] }, 2000); diff --git a/tests/tool-cache.test.mjs b/tests/tool-cache.test.mjs index 91ea9fb..adaf4dd 100644 --- a/tests/tool-cache.test.mjs +++ b/tests/tool-cache.test.mjs @@ -92,12 +92,30 @@ test("isRunnable enforces an allowlisted read-only leader", () => { assert.equal(isRunnable("sh -c 'echo hi'").ok, false); }); -test("isRunnable rejects chaining, substitution and redirects", () => { - assert.equal(isRunnable("gh api a; rm -rf /").ok, false); +test("isRunnable rejects shell operators as standalone tokens", () => { + assert.equal(isRunnable("gh api a ; rm -rf /").ok, false); assert.equal(isRunnable("gh api a && kubectl delete pod x").ok, false); - assert.equal(isRunnable("gh api $(whoami)").ok, false); assert.equal(isRunnable("gh api a > /etc/passwd").ok, false); - assert.equal(isRunnable("gh api a `id`").ok, false); + assert.equal(isRunnable("gh api a | jq .x").ok, false); // pipe belongs outside +}); + +// Regression: the old raw-string guard refused these legitimate read-only +// calls, which is what pushed a coordinator into slower workarounds. +test("isRunnable ALLOWS metacharacters inside quoted arguments", () => { + const jqSemicolon = `gh api repos/o/r/git/trees/main --jq '[.tree[].path|select(test("rcaThree";"i"))]'`; + assert.equal(isRunnable(jqSemicolon).ok, true, "; inside a jq expression is not a shell operator"); + + const urlAmp = "gh api 'search/code?q=foo&per_page=20'"; + assert.equal(isRunnable(urlAmp).ok, true, "& inside a quoted URL is not a shell operator"); + + const jqPipe = `gh pr list -R o/r --json number --jq '.[] | .number'`; + assert.equal(isRunnable(jqPipe).ok, true, "| inside a quoted jq expression is not a shell pipe"); +}); + +test("a quoted metacharacter survives tokenization as ONE literal argument", () => { + const argv = tokenize(`gh api repos/o/r --jq '[.tree[]|select(test("x";"i"))]'`); + assert.equal(argv.length, 5); + assert.equal(argv[4], '[.tree[]|select(test("x";"i"))]'); }); test("tokenize splits like a shell for quoted args, without a shell", () => { From 69aa4cd26553fbd5f1a1340b887c85d8ea3a0cca Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 19:21:22 +0530 Subject: [PATCH 13/51] fix(rca): make the tool cache work on real traffic (0% -> 26% hit rate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaying 223 recorded gh/kubectl calls from prior coordinator runs through the wrapper showed it was nearly useless in practice: only 25 calls were accepted and ZERO were hits. The cache was fine; the guard was wrong about what real traffic looks like. Three causes, each measured: - 89% of real calls embed the fetch inside a pipeline (`gh api X | jq .y`), which the guard refused outright. Now pipelines are parsed into a fetch plus a chain of pure text filters (jq/grep/head/... allowlisted), each run via execFile with no shell anywhere. Only the FETCH is keyed, so several agents filtering one fetch differently share a single network call. - `2>&1` appeared on 134 of 223 calls — agents add it reflexively because gh is chatty. It says nothing about what to fetch, and the wrapper captures stderr separately anyway, so it is normalized away instead of refused. Refusing it alone drove acceptance to zero. - `>/dev/null` was classified as "looks mutating", which is both wrong and misleading. Redirection is now separate from mutation, with an accurate message; genuine file redirects are still refused. Result on the same recorded traffic: 123/223 accepted, 32 duplicate fetches eliminated, 26% hit rate. The rest are `for repo in ...` shell loops, which legitimately need decomposing into one fetch per call — that is also better for cache granularity. Verified end-to-end that `gh api X 2>&1 | jq .a` and `... | jq .b` now share one cached fetch, and that chaining, non-filter binaries and file redirects are still refused. 123 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/cached-exec.mjs | 113 +++++++++++++++++--------------- lib/tool-cache.mjs | 134 +++++++++++++++++++++++++++++++++----- tests/tool-cache.test.mjs | 49 +++++++++++++- 3 files changed, 225 insertions(+), 71 deletions(-) diff --git a/bin/cached-exec.mjs b/bin/cached-exec.mjs index d0c6fad..ea5bcb2 100644 --- a/bin/cached-exec.mjs +++ b/bin/cached-exec.mjs @@ -68,16 +68,7 @@ if (writerOrFlag === "--stats") { process.exit(0); } -const key = cacheKey(command); -const hit = cacheGet(dir, key); - -if (hit) { - console.error(`[tool-cache HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); - process.stdout.write(hit.stdout); - process.exit(0); -} - -// Gate before running anything (allowlisted read-only leader, no chaining). +// Parse into a fetch + filter chain before anything runs. const gate = isRunnable(command); if (!gate.ok) { console.error(`[tool-cache REFUSED] ${gate.reason}`); @@ -85,51 +76,69 @@ if (!gate.ok) { process.exit(2); } -// No shell: tokenize ourselves and execFile the binary directly, so shell -// metacharacters inside arguments (a --jq expression, an XPath, a LogsQL -// filter) are passed through literally and cannot start a second command. -let argv; -try { - argv = tokenize(command); -} catch (err) { - console.error(`[tool-cache REFUSED] ${err.message}`); - process.exit(2); +// Key on the FETCH ONLY. Downstream filters are pure text transforms, so two +// agents filtering the same fetch differently share one cached network call. +const key = cacheKey(gate.fetchText); + +// Run one argv with `input` on stdin, no shell. Returns { stdout, exitCode }. +function run(argv, input) { + try { + return { + stdout: execFileSync(argv[0], argv.slice(1), { + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + // Capture stderr rather than let it inherit: execFileSync otherwise + // BOTH inherits and captures, so relaying it ourselves printed + // failures three times. + stdio: [input === undefined ? "ignore" : "pipe", "pipe", "pipe"], + ...(input === undefined ? {} : { input }), + }), + exitCode: 0, + }; + } catch (err) { + if (err.stderr) process.stderr.write(err.stderr.toString()); // the only copy + return { + stdout: (err.stdout ?? "").toString(), + exitCode: typeof err.status === "number" ? err.status : 1, + }; + } } -let stdout = ""; -let exitCode = 0; -try { - stdout = execFileSync(argv[0], argv.slice(1), { - encoding: "utf8", - maxBuffer: 64 * 1024 * 1024, - // Capture stderr instead of letting it inherit. Default execFileSync - // BOTH inherits stderr to the parent AND captures it on the error, so - // relaying `err.stderr` ourselves printed everything three times. - stdio: ["ignore", "pipe", "pipe"], - }); -} catch (err) { - // Preserve the real behaviour of the wrapped command: emit whatever it - // produced and exit non-zero. Deliberately NOT cached — a transient - // failure (rate limit, expired token) must not become a permanent answer. - stdout = (err.stdout ?? "").toString(); - exitCode = typeof err.status === "number" ? err.status : 1; - if (err.stderr) process.stderr.write(err.stderr.toString()); // now the only copy - console.error(`[tool-cache MISS ${key} — command exited ${exitCode}, NOT cached]`); - process.stdout.write(stdout); - process.exit(exitCode); +let fetched; +const hit = cacheGet(dir, key); +if (hit) { + console.error(`[tool-cache HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + fetched = hit.stdout; +} else { + const res = run(gate.fetch, undefined); + fetched = res.stdout; + if (res.exitCode !== 0) { + // Preserve the real behaviour. Deliberately NOT cached — a transient + // failure (rate limit, expired token) must not become a permanent answer. + console.error(`[tool-cache MISS ${key} — fetch exited ${res.exitCode}, NOT cached]`); + process.stdout.write(fetched); + process.exit(res.exitCode); + } + if (fetched.trim() === "") { + // An empty result is usually a wrong selector or a silently failed lookup; + // caching it creates a sticky, invisible negative for every later reader. + console.error(`[tool-cache MISS ${key} — empty result, NOT cached]`); + } else { + // nowMs is read here, at the process edge — lib/ keeps its no-clock + // discipline so it stays sandbox-safe. + cachePut(dir, key, { command: gate.fetchText, writerId: writerOrFlag, stdout: fetched, exitCode: 0 }, Date.now()); + console.error(`[tool-cache MISS ${key} — stored ${fetched.length}B]`); + } } -// An empty result is not stored. It is usually a wrong selector or a silently -// failed lookup, and caching it makes a sticky, invisible negative that every -// later reader inherits — the expensive kind of wrong. -if (stdout.trim() === "") { - console.error(`[tool-cache MISS ${key} — empty result, NOT cached]`); - process.stdout.write(stdout); - process.exit(0); +// Apply the filter chain to whatever the fetch produced (cached or fresh). +let out = fetched; +let finalExit = 0; +for (const f of gate.filters) { + const res = run(f, out); + out = res.stdout; + if (res.exitCode !== 0) { finalExit = res.exitCode; break; } } -// nowMs is read here, at the process edge, rather than inside lib/ — the -// library keeps its no-clock discipline so it stays sandbox-safe. -cachePut(dir, key, { command, writerId: writerOrFlag, stdout, exitCode }, Date.now()); -console.error(`[tool-cache MISS ${key} — stored ${stdout.length}B]`); -process.stdout.write(stdout); +process.stdout.write(out); +process.exit(finalExit); diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs index 01cf43c..4ee588d 100644 --- a/lib/tool-cache.mjs +++ b/lib/tool-cache.mjs @@ -56,7 +56,43 @@ export function cacheKey(command) { // Commands that must never be memoized, even if someone wires this into a // non-read-only context by mistake. -const MUTATING = /\b(rm|mv|cp|dd|truncate|tee)\b|>\s*\/|\bgit\s+(push|commit|merge|rebase|reset|checkout|clean)\b|\bgh\s+(pr\s+(create|merge|close|edit|comment|review)|issue\s+(create|close|edit|comment)|release\s+create|repo\s+(create|delete)|api\s+(-X\s*)?(POST|PUT|PATCH|DELETE))|\bkubectl\s+(apply|delete|edit|patch|scale|create|replace|annotate|label|cordon|drain|exec|cp|port-forward|rollout\s+(undo|restart|pause|resume))\b|\bcurl\b[^|]*\s-(X|-request)\s*(POST|PUT|PATCH|DELETE)/i; +// Note: output redirection is deliberately NOT in this list. A `>/dev/null` +// is a redirect, not a mutation, and lumping the two together produced the +// misleading refusal "command looks mutating" for ordinary read-only calls. +// Redirects are handled separately, with an accurate message. +const MUTATING = /\b(rm|mv|cp|dd|truncate|tee)\b|\bgit\s+(push|commit|merge|rebase|reset|checkout|clean)\b|\bgh\s+(pr\s+(create|merge|close|edit|comment|review)|issue\s+(create|close|edit|comment)|release\s+create|repo\s+(create|delete)|api\s+(-X\s*)?(POST|PUT|PATCH|DELETE))|\bkubectl\s+(apply|delete|edit|patch|scale|create|replace|annotate|label|cordon|drain|exec|cp|port-forward|rollout\s+(undo|restart|pause|resume))\b|\bcurl\b[^|]*\s-(X|-request)\s*(POST|PUT|PATCH|DELETE)/i; + +/** + * Strip stderr-plumbing that the wrapper already handles. + * + * `2>&1` and `2>/dev/null` appear on the majority of real recorded calls — + * agents add them reflexively because `gh` is chatty. They say nothing about + * WHAT to fetch, only where stderr should go, and the wrapper captures stderr + * separately regardless. Refusing them rejected 134 of 223 recorded calls and + * drove the effective hit rate to zero, so they are normalized away instead. + * + * Genuine FILE redirects (`> out.json`, `>> log`, `< in`) are left in place so + * the check below still refuses them: those change where data goes, which the + * wrapper cannot honour while also returning stdout to the caller. + */ +function stripStderrPlumbing(seg) { + return String(seg ?? "") + .replace(/\s*2>&1\s*/g, " ") + .replace(/\s*2>\s*\/dev\/null\s*/g, " ") + .trim(); +} + +/** True if the segment still contains a real redirect outside quotes, after + * stderr plumbing has been normalized away. */ +function hasUnquotedRedirect(seg) { + let quote = null; + for (const ch of String(seg ?? "")) { + if (quote) { if (ch === quote) quote = null; continue; } + if (ch === "'" || ch === '"') { quote = ch; continue; } + if (ch === ">" || ch === "<") return true; + } + return false; +} export function isCacheable(command) { return !MUTATING.test(String(command ?? "")); @@ -66,6 +102,37 @@ export function isCacheable(command) { // refused outright. const ALLOWED_LEADER = /^\s*(gh|kubectl|curl|git)\s/; +// Pure text filters allowed AFTER the fetch in a pipeline. They transform the +// fetch's output and never reach the network, so they are deliberately outside +// the cache key: `gh api X | jq .a` and `gh api X | jq .b` share ONE cached +// fetch. Measured motivation — replaying real recorded traffic, 89% of calls +// embedded the fetch in a pipeline, so refusing pipelines outright meant the +// cache applied to almost nothing in practice. +const ALLOWED_FILTER = new Set([ + "jq", "grep", "egrep", "head", "tail", "sort", "uniq", "wc", + "cut", "tr", "sed", "awk", "base64", "python3", "rev", "column", +]); + +/** Split on top-level `|` only — a pipe inside quotes (a jq expression) stays + * part of its segment. Returns trimmed segment strings. */ +export function splitPipeline(command) { + const segs = []; + let cur = ""; + let quote = null; + for (const ch of String(command ?? "")) { + if (quote) { + cur += ch; + if (ch === quote) quote = null; + continue; + } + if (ch === "'" || ch === '"') { quote = ch; cur += ch; continue; } + if (ch === "|") { segs.push(cur.trim()); cur = ""; continue; } + cur += ch; + } + segs.push(cur.trim()); + return segs.filter((s) => s.length > 0); +} + // Shell operators, checked as whole ARGV TOKENS rather than by scanning the // raw string. Scanning the string was too blunt and refused legitimate reads: // `--jq 'test("rcaThree";"i")'` was rejected for the `;` inside a quoted jq @@ -88,28 +155,63 @@ const OPERATOR_TOKENS = new Set([";", "|", "||", "&&", "&", ">", ">>", "<", "<<" * and refusing `;`-chained loops nudges callers toward one-fetch-per-call, * which caches far better anyway. */ +/** + * Validate a command and return its execution plan. + * + * On success: `{ ok, fetch: string[], filters: string[][] }` — the fetch is + * executed (or served from cache) and its output is piped through the filters, + * each run via execFile with NO shell anywhere in the chain. + * + * Pipelines are accepted rather than refused because refusing them is what + * made the cache useless on real traffic. Only the FETCH is keyed, so several + * agents filtering one fetch differently all share a single cached result. + */ export function isRunnable(command) { const c = String(command ?? ""); - if (!ALLOWED_LEADER.test(c)) { + if (!isCacheable(c)) return { ok: false, reason: "command looks mutating" }; + + const segments = splitPipeline(c).map(stripStderrPlumbing).filter((s) => s.length > 0); + if (segments.length === 0) return { ok: false, reason: "empty command" }; + + if (!ALLOWED_LEADER.test(segments[0])) { return { ok: false, reason: "command must start with gh, kubectl, curl, or git" }; } - if (!isCacheable(c)) return { ok: false, reason: "command looks mutating" }; - let argv; - try { - argv = tokenize(c); - } catch (err) { - return { ok: false, reason: err.message }; + const parsed = []; + for (const seg of segments) { + if (hasUnquotedRedirect(seg)) { + return { + ok: false, + reason: "file redirects (>, >>, <) are not supported — the wrapper returns stdout to you directly, so drop the redirect. (`2>&1` and `2>/dev/null` are fine; they're stripped automatically.)", + }; + } + let argv; + try { + argv = tokenize(seg); + } catch (err) { + return { ok: false, reason: err.message }; + } + if (argv.length === 0) return { ok: false, reason: "empty pipeline segment" }; + const op = argv.find((t) => OPERATOR_TOKENS.has(t)); + if (op) { + return { + ok: false, + reason: `'${op}' is a shell operator. Pipes are supported, but ';', '&&', redirects and substitution are not — issue one fetch per call.`, + }; + } + parsed.push(argv); } - const op = argv.find((t) => OPERATOR_TOKENS.has(t)); - if (op) { - return { - ok: false, - reason: `'${op}' is a shell operator — run it OUTSIDE the wrapper (e.g. \`cached-exec … 'gh api X' | jq .y\`) so one cached fetch can serve several different filters. Metacharacters inside a quoted argument are fine.`, - }; + + for (const argv of parsed.slice(1)) { + if (!ALLOWED_FILTER.has(argv[0])) { + return { + ok: false, + reason: `'${argv[0]}' is not an allowed filter after the fetch (allowed: ${[...ALLOWED_FILTER].join(", ")})`, + }; + } } - if (argv.length === 0) return { ok: false, reason: "empty command" }; - return { ok: true }; + + return { ok: true, fetch: parsed[0], filters: parsed.slice(1), fetchText: segments[0] }; } /** diff --git a/tests/tool-cache.test.mjs b/tests/tool-cache.test.mjs index adaf4dd..e8e01e2 100644 --- a/tests/tool-cache.test.mjs +++ b/tests/tool-cache.test.mjs @@ -5,7 +5,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { toolCacheDirFor, cacheKey, mcpCacheKey, cacheGet, cachePut, cacheStats, - isCacheable, isCacheableMcp, isRunnable, redact, tokenize, + isCacheable, isCacheableMcp, isRunnable, redact, tokenize, splitPipeline, } from "../lib/tool-cache.mjs"; let dir; @@ -92,11 +92,54 @@ test("isRunnable enforces an allowlisted read-only leader", () => { assert.equal(isRunnable("sh -c 'echo hi'").ok, false); }); -test("isRunnable rejects shell operators as standalone tokens", () => { +test("isRunnable rejects chaining and redirects, but ACCEPTS pipelines", () => { assert.equal(isRunnable("gh api a ; rm -rf /").ok, false); assert.equal(isRunnable("gh api a && kubectl delete pod x").ok, false); assert.equal(isRunnable("gh api a > /etc/passwd").ok, false); - assert.equal(isRunnable("gh api a | jq .x").ok, false); // pipe belongs outside + // `2>&1` is stderr plumbing the wrapper already owns — stripped, not refused. + // Refusing it rejected 134 of 223 real recorded calls and zeroed the hit rate. + assert.equal(isRunnable("gh api a 2>&1").ok, true, "stderr plumbing is normalized away"); + assert.equal(isRunnable("gh api a 2>/dev/null | jq .x").ok, true); + // Pipelines are supported now: refusing them meant the cache applied to + // almost no real traffic, since most fetches are written inline with a filter. + assert.equal(isRunnable("gh api a | jq .x").ok, true); +}); + +test("a FILE redirect is reported as a redirect, not as a mutation", () => { + const r = isRunnable("gh api repos/x > out.json"); + assert.equal(r.ok, false); + assert.match(r.reason, /redirect/i); + assert.doesNotMatch(r.reason, /mutating/i); +}); + +test("stderr plumbing does not change the cache key", () => { + const a = isRunnable("gh api repos/x | jq .a"); + const b = isRunnable("gh api repos/x 2>&1 | jq .b"); + assert.equal(cacheKey(a.fetchText), cacheKey(b.fetchText)); +}); + +test("pipeline plan: only the FETCH is keyed, filters are separate", () => { + const a = isRunnable("gh api repos/x | jq -r .name"); + const b = isRunnable("gh api repos/x | jq -r .branch | tr a-z A-Z"); + assert.equal(a.ok && b.ok, true); + // Same underlying fetch -> same cache key -> one network call serves both. + assert.equal(cacheKey(a.fetchText), cacheKey(b.fetchText)); + assert.deepEqual(a.fetch, ["gh", "api", "repos/x"]); + assert.equal(a.filters.length, 1); + assert.equal(b.filters.length, 2); +}); + +test("only pure text filters may follow the fetch", () => { + assert.equal(isRunnable("gh api repos/x | jq .a").ok, true); + assert.equal(isRunnable("gh api repos/x | grep foo").ok, true); + assert.equal(isRunnable("gh api repos/x | sh").ok, false); + assert.equal(isRunnable("gh api repos/x | bash -c 'x'").ok, false); + assert.equal(isRunnable("gh api repos/x | kubectl delete pod y").ok, false); +}); + +test("splitPipeline ignores a pipe inside quotes", () => { + assert.deepEqual(splitPipeline(`gh pr list --jq '.[] | .number' | head -5`), + ["gh pr list --jq '.[] | .number'", "head -5"]); }); // Regression: the old raw-string guard refused these legitimate read-only From 9e14eecf1bf339d43856c074e30a07ca2e408694 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 19:41:44 +0530 Subject: [PATCH 14/51] fix(rca): three defects found by a 6-coordinator swarm on a live build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dispatched six coordinators concurrently against a warm cache. Five of six resolved; the swarm found three real defects, one of them data-destroying. 1. DATA LOSS: redaction consumed the rest of the LINE after a secret-ish key. GitHub's file API returns SINGLE-LINE JSON whose download_url always carries `?token=…`, so a 214,074-byte response was cached as 816 bytes with the content field silently gone. Every private-repo file fetch through the cache was corrupted, and a coordinator hit exactly the predicted failure: it could not verify a line-number-specific hypothesis and nearly falsified a claim that was in fact TRUE (TestRunService.java:1747 really is an unguarded `.get(0)`). Redaction is now bounded by the first structural delimiter. Two already-corrupted entries were purged from the live cache. 2. Neither tokenize nor splitPipeline handled backslash escapes, so `\"` read as a closing quote. Two coordinators hit the two symptoms: jq string equality arrived mangled (`select(.filename==\a/b.json\)`), and a `|` in a jq regex alternation was split as a shell pipe, refusing the command. One fix, both symptoms — escapes are now POSIX-style (literal everywhere except inside single quotes). 3. Two findings corroborated independently by multiple coordinators, now in the contract: a drain error kills the TURN, not the THREAD — resubmitting on the same threadId resolves immediately, whereas the old reading ended PENDING and discarded a resolvable test; and the turn-2 wedge correlates with message size (~1400/~1350-char submits failed, a ~940-char retry landed) against a configured 1000-char cap. Cache went 33 -> 63 entries over the swarm with 7 cross-agent hits, all seven writer shards present and no collisions. 128 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 17 +++++++++++ lib/tool-cache.mjs | 55 ++++++++++++++++++++++++++++++------ tests/tool-cache.test.mjs | 44 +++++++++++++++++++++++++++++ 3 files changed, 107 insertions(+), 9 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 163fe0d..a52b74d 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -165,6 +165,23 @@ read-only and has no side effects, so a read is always safe to repeat. extra turn, never a busy-wait. 4. **One thread per test.** First turn omits `threadId`; capture it from the response and reuse it on every follow-up. Never start a second thread. +4b. **A drain ERROR kills the TURN, not the THREAD — resubmit, don't give up.** + `getTfaTurnResult` returning `TFA agent run failed` (or the submit itself + throwing it) is a dead turn, not a dead thread: observed repeatedly, a + fresh submit on the SAME `threadId` succeeds immediately and resolves at + high confidence. So when the drain fast-fails on consecutive hard errors, + the next move is to resubmit on that same thread (counting it as a turn) — + NOT to mint a new thread and not to end the run `PENDING`. Ending PENDING + here throws away a resolvable test. Only stop once the turn cap is spent. + +4c. **Keep every turn message under `turnMessageMaxChars` (1000).** The + turn-2 wedge above correlates with message size: submits of ~1400 and + ~1350 chars failed back-to-back on one thread while a ~940-char retry was + accepted and resolved. Treat the configured cap as a hard budget, not a + soft target — trim the digest (drop `low`-priority blocks first, link + instead of quoting) rather than sending an oversized message and burning + turns on a failure that looks like a server fault. + 5. **Soft-PENDING is DRAINED, not reported.** `status: "PENDING"` means the tool's 90s in-call poll expired, not that TFA has nothing to say — turns landing past 90s are routine (a first turn finalizing `NEEDS_INFO` at 104s is a real, diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs index 4ee588d..e77f71f 100644 --- a/lib/tool-cache.mjs +++ b/lib/tool-cache.mjs @@ -119,7 +119,18 @@ export function splitPipeline(command) { const segs = []; let cur = ""; let quote = null; - for (const ch of String(command ?? "")) { + const s = String(command ?? ""); + for (let i = 0; i < s.length; i++) { + const ch = s[i]; + // A backslash escapes the next character. Without this, `\"` inside a + // double-quoted jq filter reads as "close quote", the parser thinks it is + // back outside quotes, and a `|` in a regex alternation like + // `test("vite|env";"i")` gets split as a shell pipe. + if (ch === "\\" && i + 1 < s.length && quote !== "'") { + cur += ch + s[i + 1]; + i++; + continue; + } if (quote) { cur += ch; if (ch === quote) quote = null; @@ -130,7 +141,7 @@ export function splitPipeline(command) { cur += ch; } segs.push(cur.trim()); - return segs.filter((s) => s.length > 0); + return segs.filter((s2) => s2.length > 0); } // Shell operators, checked as whole ARGV TOKENS rather than by scanning the @@ -226,7 +237,19 @@ export function tokenize(command) { let cur = ""; let quote = null; let started = false; - for (const ch of String(command ?? "")) { + const s = String(command ?? ""); + for (let i = 0; i < s.length; i++) { + const ch = s[i]; + // Backslash escape, POSIX-style: literal everywhere except inside single + // quotes. Missing this mangled jq's most common idiom — `\"` was treated + // as a quote delimiter, so `select(.filename==\"x\")` reached the binary + // as `select(.filename==\x\)` with the quotes eaten. + if (ch === "\\" && i + 1 < s.length && quote !== "'") { + cur += s[i + 1]; + i++; + started = true; + continue; + } if (quote) { if (ch === quote) quote = null; else cur += ch; @@ -271,17 +294,31 @@ export function mcpCacheKey(toolName, args) { return createHash("sha256").update(payload).digest("hex").slice(0, 24); } -// Redact the REST OF THE LINE after a secret-ish key, not just the next -// token: `Authorization: Bearer <tok>` puts the actual credential in the -// second word, so a `\S+` capture would leave it sitting on disk. -const SECRET_LINE = - /((?:token|authorization|api[_-]?key|secret|password|passwd|bearer|access[_-]?key)\s*[=:]\s*)([^\r\n]*)/gi; +// Redact the secret VALUE, bounded by the first structural delimiter — never +// "the rest of the line". +// +// The rest-of-line version silently destroyed data. GitHub's file API returns +// SINGLE-LINE JSON whose `download_url` always carries `?token=…`, so matching +// `token=` and consuming `[^\r\n]*` swallowed the entire remaining payload: +// a 214KB response cached as 816 bytes, content field gone, no warning. Every +// private-repo file fetch was affected. +// +// A value therefore stops at whitespace, quote, comma, semicolon, brace, +// bracket or `&` — enough to cover a real credential, never enough to eat the +// surrounding document. +const SECRET_KV = + /((?:token|authorization|api[_-]?key|secret|password|passwd|access[_-]?key)"?\s*[=:]\s*"?)((?:bearer|basic|token)\s+)?([^\s"'`,;}\]&\r\n]{4,})/gi; + +// A bare `Bearer <token>` / `Basic <token>` with no key= in front of it. +const SECRET_SCHEME = /\b(bearer|basic)\s+([A-Za-z0-9._~+/=-]{8,})/gi; /** Redact anything token-shaped before it is persisted. The cache lives in * temp, but a cached `gh api` response or log line could still carry a * credential, and "it's only temp" is not a reason to write one to disk. */ export function redact(text) { - return String(text ?? "").replace(SECRET_LINE, (_m, k) => `${k}<redacted>`); + return String(text ?? "") + .replace(SECRET_KV, (_m, key) => `${key}<redacted>`) + .replace(SECRET_SCHEME, (_m, scheme) => `${scheme} <redacted>`); } const MAX_BYTES = 256 * 1024; diff --git a/tests/tool-cache.test.mjs b/tests/tool-cache.test.mjs index e8e01e2..56e22da 100644 --- a/tests/tool-cache.test.mjs +++ b/tests/tool-cache.test.mjs @@ -59,6 +59,50 @@ test("redact leaves ordinary output untouched", () => { assert.equal(redact("just some log output"), "just some log output"); }); +// Regression, found by a live coordinator. Redaction used to consume the REST +// OF THE LINE after a secret-ish key. GitHub's file API returns SINGLE-LINE +// JSON whose download_url always carries `?token=…`, so a 214KB response was +// silently cached as 816 bytes with the content field gone — every +// private-repo file fetch was corrupted, with no warning. +test("redact bounds the value and does NOT eat the rest of a single-line JSON", () => { + const json = '{"name":"F.java","download_url":"https://raw.example/F.java?token=BRFIJBHPIG5IILHZ",' + + '"type":"file","content":"' + "A".repeat(5000) + '"}'; + const out = redact(json); + assert.ok(!out.includes("BRFIJBHPIG5IILHZ"), "the token itself must be redacted"); + assert.ok(out.includes('"type":"file"'), "structure after the token must survive"); + assert.ok(out.includes("A".repeat(5000)), "the content payload must survive"); + assert.ok(out.length > 5000, `expected full payload, got ${out.length} bytes`); +}); + +test("redact still catches a bare Bearer token and a key=value secret", () => { + assert.equal(redact("Authorization: Bearer abc123SECRET"), "Authorization: <redacted>"); + assert.equal(redact("api_key=zzz999"), "api_key=<redacted>"); + assert.ok(!redact("Bearer eyJhbGciOiJIUzI1NiJ9").includes("eyJhbGciOiJIUzI1NiJ9")); +}); + +// Regression, found by two live coordinators. Neither tokenize nor +// splitPipeline handled backslash escapes, so `\"` read as a closing quote. +// That mangled jq's two most common idioms: string equality and, because the +// parser then believed it was outside quotes, regex alternation got split as +// a shell pipe. +test("escaped double quotes survive tokenization for jq", () => { + const argv = tokenize(String.raw`gh api x --jq .[]|select(.filename==\"a/b.json\")`); + assert.equal(argv[argv.length - 1], '.[]|select(.filename=="a/b.json")'); +}); + +test("a pipe inside an escaped-quote jq regex is NOT a shell pipe", () => { + const cmd = String.raw`gh pr view 51044 --json files | jq -c "select(test(\"vite|env|s3\";\"i\"))"`; + assert.deepEqual(splitPipeline(cmd).length, 2, "must split into fetch + one filter only"); + const g = isRunnable(cmd); + assert.equal(g.ok, true, g.reason); + assert.deepEqual(g.fetch, ["gh", "pr", "view", "51044", "--json", "files"]); + assert.equal(g.filters[0][2], 'select(test("vite|env|s3";"i"))'); +}); + +test("single quotes suppress escape processing, POSIX-style", () => { + assert.deepEqual(tokenize(String.raw`gh api 'a\nb'`), ["gh", "api", String.raw`a\nb`]); +}); + test("oversized payloads are truncated and flagged", () => { const k = cacheKey("gh api big"); const rec = cachePut(dir, k, { command: "gh api big", stdout: "x".repeat(400 * 1024) }, 1000); From 9955ba3a9862276e6252600e01f385323f3f815c Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 19:56:07 +0530 Subject: [PATCH 15/51] fix(rca): flip() silently discarded results written in RCA_OUTPUT vocabulary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `flip` accepted only the lowercase CSV states and returned a bare `false` for anything else — including `RESOLVED`, which is the exact value the RCA_OUTPUT contract tells every coordinator to emit. Callers that didn't check the return saw no difference between success and a no-op, so their rows stayed `pending` and looked un-run. This was not theoretical: a six-coordinator swarm appeared to "skip" the CSV flip entirely. A later coordinator diagnosed the real cause — they had almost certainly called flip with `RESOLVED` and been silently refused. Field names had the same trap: `thread_id`/`status` were dropped without comment by the COLUMNS guard, since the columns are `threadId`/`rca_done`. Accept and normalize the output-block vocabulary (RESOLVED->resolved, PENDING->pending-resume, case-insensitive) plus the field aliases, and make genuine rejections and dropped keys warn loudly instead of failing silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/csv-state.mjs | 50 +++++++++++++++++++++++++++++++++++++--- tests/csv-state.test.mjs | 22 ++++++++++++++++++ 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 724eea3..8590a66 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -221,20 +221,64 @@ export function heartbeat(csvPath, testRunId, worker, nowMs) { // in-flight claim. `fields` carries any of: rca_done, root_cause, failure_type, // possible_fix, related_prs, threadId, turnId, coverage, confidence, // last_evidence_digest, cluster_id. +// The RCA_OUTPUT contract speaks `RESOLVED | PENDING | failed`, while the CSV +// stores `resolved | blocked | failed | pending-resume`. Callers naturally pass +// the vocabulary their own output block mandates, so accept it and translate +// rather than silently rejecting — a silent `false` here cost a whole batch of +// results, since the row simply stayed `pending` and looked un-run. +const FLIP_ALIASES = new Map([ + ["resolved", "resolved"], + ["pending", RESUMABLE], + ["pending-resume", RESUMABLE], + ["blocked", "blocked"], + ["failed", "failed"], + ["done", "resolved"], +]); + +// Column aliases for the same reason: the output block says `thread_id` and +// `status`, the CSV says `threadId` and `rca_done`. +const COLUMN_ALIASES = new Map([ + ["thread_id", "threadId"], + ["turn_id", "turnId"], + ["status", "rca_done"], + ["test_run_id", "testRunId"], +]); + export function flip(csvPath, testRunId, fields, nowMs) { // Enforce the contract: a flip must name a valid outcome. A partial flip with // a missing/non-terminal rca_done would otherwise clear the claim yet leave the // row `pending` — re-exposing it for a duplicate RCA that clobbers this result. // Reject without mutating so the worker keeps its claim and the bug surfaces. - if (!FLIP_STATES.has(fields?.rca_done)) return false; + const raw = fields?.rca_done ?? fields?.status; + const state = FLIP_ALIASES.get(String(raw ?? "").trim().toLowerCase()); + if (!state) { + // Loud, not silent: the previous bare `false` was indistinguishable from + // success to a caller that didn't check, and results were lost that way. + console.warn( + `[csv-state] flip REJECTED for testRunId=${testRunId}: rca_done=${JSON.stringify(raw)} ` + + `is not one of ${[...new Set(FLIP_ALIASES.values())].join(" | ")} (case-insensitive). Row NOT written.`, + ); + return false; + } const rows = readRows(csvPath); const row = rows.find((r) => String(r.testRunId) === String(testRunId)); - if (!row) return false; - for (const [k, v] of Object.entries(fields)) { + if (!row) { + console.warn(`[csv-state] flip REJECTED: no row for testRunId=${testRunId} in ${csvPath}`); + return false; + } + const dropped = []; + for (const [k0, v] of Object.entries(fields)) { + const k = COLUMN_ALIASES.get(k0) ?? k0; if (COLUMNS.includes(k)) { row[k] = Array.isArray(v) ? v.join("; ") : (v ?? ""); + } else { + dropped.push(k0); } } + row.rca_done = state; // normalized, whatever spelling arrived + if (dropped.length) { + console.warn(`[csv-state] flip ignored unknown field(s) for ${testRunId}: ${dropped.join(", ")}`); + } row.in_flight_worker = ""; row.timestamp = String(nowMs); writeRows(csvPath, rows); diff --git a/tests/csv-state.test.mjs b/tests/csv-state.test.mjs index 3cbc83a..b5caeb3 100644 --- a/tests/csv-state.test.mjs +++ b/tests/csv-state.test.mjs @@ -121,6 +121,28 @@ test("pendingRows returns only pending work", () => { assert.equal(pend[0].testRunId, "102"); }); +// Regression: `flip` used to accept ONLY the lowercase CSV vocabulary and +// return a bare `false` for anything else — including `RESOLVED`, the exact +// value the RCA_OUTPUT contract mandates. A whole batch of coordinator results +// was lost that way: they called flip, got a silent no-op, and the rows stayed +// `pending` looking un-run. +test("flip accepts the RCA_OUTPUT vocabulary and normalizes it", () => { + seed(csv, "build-1", TESTS); + assert.equal(flip(csv, 101, { rca_done: "RESOLVED", root_cause: "x" }, 1000), true); + assert.equal(readRows(csv).find((r) => r.testRunId === "101").rca_done, "resolved"); + + assert.equal(flip(csv, 102, { status: "PENDING" }, 1000), true); + assert.equal(readRows(csv).find((r) => r.testRunId === "102").rca_done, "pending-resume"); +}); + +test("flip maps the output block's field names onto real columns", () => { + seed(csv, "build-1", TESTS); + flip(csv, 101, { rca_done: "resolved", thread_id: "chat:101", turn_id: "t-7" }, 1000); + const row = readRows(csv).find((r) => r.testRunId === "101"); + assert.equal(row.threadId, "chat:101"); + assert.equal(row.turnId, "t-7"); +}); + test("flip rejects a missing/non-terminal rca_done without mutating the row", () => { seed(csv, "build-1", TESTS); claim(csv, 101, "w1", 1000); From e088c4bb629dd8e48623beaf1913b9ad32a5f12f Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:04:58 +0530 Subject: [PATCH 16/51] fix(rca): retract the message-size wedge claim; surface folded evidence and hit counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from a warm-cache verification run. 1. RETRACTION. The contract told coordinators to keep turn messages under 1000 chars to avoid the TFA wedge, on the strength of a two-point correlation (~1400/~1350-char submits failing where a ~940-char retry landed). A later run refuted it: a 240-char message wedged exactly as a 1500-char one did. The cap stays — a tight digest is the contract — but it is no longer presented as a wedge cure, and a wedge must not be read as evidence the message was too long. The reliable response remains resubmit on the same thread. 2. Contributions were invisible to the obvious read. Coordinators are handed the base path and naturally `cat` it, which shows base only — one agent reported "2 repos" when the folded view had 5, including an 11-PR observability-api entry a sibling had contributed. Shards are what make concurrent write-back safe, so add `bin/evidence-show.mjs` to print the merged view (with --summary flagging repos whose PR list was never actually searched) rather than give up the layout. 3. Hit-rate was unmeasurable in normal use: callers silence tool chatter with `2>/dev/null`, which also discards the banner the metric depends on. Banners now tee to TOOLCACHE_LOG when set, so stderr can be suppressed and hits still counted. Warm-cache result that motivated this: cross-agent hits confirmed — one coordinator re-filtered a sibling's cached 76KB PR diff three ways for free, another reused two cached diffs it never fetched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 17 ++++++---- bin/cached-exec.mjs | 33 +++++++++++++++---- bin/evidence-show.mjs | 64 ++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+), 14 deletions(-) create mode 100644 bin/evidence-show.mjs diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index a52b74d..afe92c4 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -174,13 +174,16 @@ read-only and has no side effects, so a read is always safe to repeat. NOT to mint a new thread and not to end the run `PENDING`. Ending PENDING here throws away a resolvable test. Only stop once the turn cap is spent. -4c. **Keep every turn message under `turnMessageMaxChars` (1000).** The - turn-2 wedge above correlates with message size: submits of ~1400 and - ~1350 chars failed back-to-back on one thread while a ~940-char retry was - accepted and resolved. Treat the configured cap as a hard budget, not a - soft target — trim the digest (drop `low`-priority blocks first, link - instead of quoting) rather than sending an oversized message and burning - turns on a failure that looks like a server fault. +4c. **Keep every turn message under `turnMessageMaxChars` (1000)** — for + digest discipline, NOT as a wedge cure. An early correlation suggested + oversized messages caused the turn wedge (~1400/~1350-char submits failed + where a ~940-char retry landed, twice), but a later run refuted it + outright: a 240-char message wedged exactly as a 1500-char one did. So + respect the cap because a tight digest is the contract (link, don't paste) + — but do not expect trimming to prevent a wedge, and do not read a wedge + as evidence your message was too long. The wedge is a TFA-side fault whose + trigger is still unidentified; the reliable response is 4b (resubmit on the + same thread), not shrinking the payload. 5. **Soft-PENDING is DRAINED, not reported.** `status: "PENDING"` means the tool's 90s in-call poll expired, not that TFA has nothing to say — turns landing past diff --git a/bin/cached-exec.mjs b/bin/cached-exec.mjs index ea5bcb2..b6ab9fd 100644 --- a/bin/cached-exec.mjs +++ b/bin/cached-exec.mjs @@ -21,8 +21,9 @@ // 1. Hit/miss banners go to STDERR, so stdout stays byte-identical to the raw // command and `| jq` works. But `2>&1 | jq` merges the banner back into // the pipe and jq dies on it ("Invalid literal at line 1, column 12"). -// Don't redirect stderr into a pipe; if you must silence it, `2>/dev/null` -// — though that also hides whether you got a hit. +// Don't redirect stderr into a pipe. If you silence it with `2>/dev/null` +// you also lose the hit/miss signal — so set `TOOLCACHE_LOG=<path>` and +// the banners are teed there too: `grep -c HIT <path>` still works. // // 2. Nested single quotes. A command containing its own `'…'` (typically // `--jq '.[] | "\(.number)"'`) cannot be passed inside a single-quoted @@ -32,7 +33,7 @@ // | node bin/cached-exec.mjs "$B" 3895 - import { execFileSync } from "node:child_process"; -import { readFileSync } from "node:fs"; +import { readFileSync, appendFileSync } from "node:fs"; import { toolCacheDirFor, cacheKey, cacheGet, cachePut, cacheStats, isRunnable, tokenize, } from "../lib/tool-cache.mjs"; @@ -62,6 +63,24 @@ if (!buildId || (writerOrFlag !== "--stats" && !command)) { const dir = toolCacheDirFor(buildId, process.env.RCA_STATE_DIR ?? ""); +// Where hit/miss banners go. Default stderr keeps stdout byte-identical to the +// wrapped command. But callers pipe stdout into jq/sed and silence stderr with +// `2>/dev/null` to keep the tool chatter out — which also throws away the +// banner, so the run's own hit-rate becomes unmeasurable. Setting +// TOOLCACHE_LOG=<path> tees banners to a file, letting a caller suppress +// stderr and still count hits afterwards (`grep -c HIT <path>`). +const logPath = process.env.TOOLCACHE_LOG ?? ""; +function banner(line) { + console.error(line); + if (logPath) { + try { + appendFileSync(logPath, line + "\n", { encoding: "utf8", mode: 0o600 }); + } catch { + /* logging must never break the fetch */ + } + } +} + if (writerOrFlag === "--stats") { const s = cacheStats(dir); console.log(JSON.stringify({ cacheDir: dir, ...s }, null, 2)); @@ -107,7 +126,7 @@ function run(argv, input) { let fetched; const hit = cacheGet(dir, key); if (hit) { - console.error(`[tool-cache HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + banner(`[tool-cache HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); fetched = hit.stdout; } else { const res = run(gate.fetch, undefined); @@ -115,19 +134,19 @@ if (hit) { if (res.exitCode !== 0) { // Preserve the real behaviour. Deliberately NOT cached — a transient // failure (rate limit, expired token) must not become a permanent answer. - console.error(`[tool-cache MISS ${key} — fetch exited ${res.exitCode}, NOT cached]`); + banner(`[tool-cache MISS ${key} — fetch exited ${res.exitCode}, NOT cached]`); process.stdout.write(fetched); process.exit(res.exitCode); } if (fetched.trim() === "") { // An empty result is usually a wrong selector or a silently failed lookup; // caching it creates a sticky, invisible negative for every later reader. - console.error(`[tool-cache MISS ${key} — empty result, NOT cached]`); + banner(`[tool-cache MISS ${key} — empty result, NOT cached]`); } else { // nowMs is read here, at the process edge — lib/ keeps its no-clock // discipline so it stays sandbox-safe. cachePut(dir, key, { command: gate.fetchText, writerId: writerOrFlag, stdout: fetched, exitCode: 0 }, Date.now()); - console.error(`[tool-cache MISS ${key} — stored ${fetched.length}B]`); + banner(`[tool-cache MISS ${key} — stored ${fetched.length}B]`); } } diff --git a/bin/evidence-show.mjs b/bin/evidence-show.mjs new file mode 100644 index 0000000..37d7bc1 --- /dev/null +++ b/bin/evidence-show.mjs @@ -0,0 +1,64 @@ +#!/usr/bin/env node +// Print the FOLDED evidence view: the orchestrator's base pre-fetch with every +// coordinator's contribution shard merged on top. +// +// Why this exists: coordinators are handed one path — the base file — and +// naturally read it with `cat`/`jq`. That shows base ONLY, so every +// contribution written by a sibling is invisible. A real run hit this: an +// agent reported "the file has 2 repos" when the folded view had 5, including +// the 11-PR observability-api entry a prior coordinator had contributed. The +// shard layout is what makes concurrent write-back safe, so the fix is to give +// the merged view its own command rather than to abandon shards. +// +// Usage: +// node bin/evidence-show.mjs <evidenceFilePath> # full folded JSON +// node bin/evidence-show.mjs <evidenceFilePath> --summary # one line per repo/workload +// node bin/evidence-show.mjs <evidenceFilePath> --repo <name> + +import { readEvidenceFile, readBaseFile, contribDirFor, hasTrustworthyPrList } from "../lib/evidence-file.mjs"; +import { existsSync, readdirSync } from "node:fs"; + +const [, , filePath, mode, arg] = process.argv; +if (!filePath) { + console.error("usage: evidence-show.mjs <evidenceFilePath> [--summary | --repo <name>]"); + process.exit(2); +} + +const folded = readEvidenceFile(filePath); + +if (mode === "--repo") { + console.log(JSON.stringify(folded.github?.[arg] ?? null, null, 2)); + process.exit(0); +} + +if (mode !== "--summary") { + console.log(JSON.stringify(folded, null, 2)); + process.exit(0); +} + +const base = readBaseFile(filePath); +const dir = contribDirFor(filePath); +const shards = existsSync(dir) ? readdirSync(dir).filter((f) => f.endsWith(".json")) : []; + +console.log(`build : ${folded.buildId}`); +console.log(`base repos : ${Object.keys(base.github ?? {}).length}`); +console.log(`contribution shards: ${shards.length} (${shards.map((s) => s.replace(".json", "")).join(", ") || "none"})`); +console.log(""); +console.log("github (folded):"); +for (const [repo, e] of Object.entries(folded.github ?? {})) { + const prs = (e.prsInWindow ?? []).length; + const trust = hasTrustworthyPrList(folded, repo) ? "trustworthy" : "PR LIST NOT TRUSTWORTHY (never searched)"; + console.log(` ${repo}: ${prs} PR(s), deployState=${e.deployState ? "yes" : "no"}, gap=${e.gap ?? "none"} — ${trust}`); +} +console.log(""); +console.log("logs (folded):"); +for (const [wl, e] of Object.entries(folded.logs ?? {})) { + const k = e.kubectlSweep?.gap ? "gapped" : e.kubectlSweep ? "present" : "absent"; + const v = e.victorialogs?.gap ? "gapped" : e.victorialogs ? "present" : "absent"; + console.log(` ${wl}: kubectl=${k}, victorialogs=${v}, gap=${e.gap ?? "none"}`); +} +if (folded.coverage?.reposWithUntrustedPrList?.length) { + console.log(""); + console.log(`WARNING untrusted PR lists: ${folded.coverage.reposWithUntrustedPrList.join(", ")}`); + console.log(" an empty prsInWindow here does NOT mean 'no PRs' — search live before concluding."); +} From c91f7c906c67d068a96e69dccd4d1548b19f1f12 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:16:22 +0530 Subject: [PATCH 17/51] docs(rca): require pre-seeding the MCP cache from Step 4's own queries The MCP cache had stored zero entries across every live run. The cause is structural: an agent's get -> call -> put costs three tool calls on a miss to save one later, so skipping it is correct for a one-off query, and every coordinator did. Pre-seeding inverts the economics. The orchestrator already runs these log sweeps in Step 4, so depositing each digest under the key a coordinator would compute makes the agent's `get` a single call that usually hits. Verified: after seeding four VictoriaLogs digests, a coordinator get returns HIT, and arg-order canonicalization means a differently-ordered query still matches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- skills/rca-build/SKILL.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index a4e6632..e366b34 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -349,6 +349,21 @@ that agent type — no need to repeat the mechanics in the prompt, just don't omit `evidenceFilePath` (above), since write-back has nothing to write to without it. +**Pre-seed the MCP cache with the queries you just ran.** Step 4's log sweeps +are MCP calls, and a coordinator will often want the same ones. Deposit each +result under the key it would compute — `mcpCacheKey(tool, args)` then +`cachePut(toolCacheDirFor(buildId), key, {…, writerId: "orchestrator"}, nowMs)` +from `lib/tool-cache.mjs` — storing the DIGEST, not the raw rows. + +This is not optional polish; without it the MCP cache goes unused. Measured +across every live run before this was added: **zero MCP entries ever stored**, +because an agent's check-then-call-then-store costs three calls on a miss to +save one later, so skipping it is the rational choice for a one-off query. +Pre-seeding inverts that — the agent's `get` is a single call that usually +hits. Store the same digest you put in the evidence file; the two are +complementary (the file is read wholesale at turn 1, the cache answers a +specific repeat query later). + **Also hand every dispatch the tool cache.** The evidence file shares digested *findings*; `bin/cached-exec.mjs` / `bin/cached-mcp.mjs` share raw *call results*, which is where most duplicate work actually hides — on one measured From 3c55f1f8c01761945a908f2956f85cdf3ffacad8 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:27:42 +0530 Subject: [PATCH 18/51] fix(rca): three gaps a warm-system verification run exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. `flip` silently dropped `view_rca` and `turns_used` — both are in the RCA_OUTPUT contract but had no CSV column, and `view_rca` is the dashboard link the entire run exists to produce. Added as columns. 2. An MCP cache HIT requires reproducing the args exactly, and canonicalization normalizes key ORDER, not content — so a coordinator that guesses the logql/window/limit triple misses. One run burned four probe calls guessing to save two, making the cache a net loss on that pass. Added `cached-mcp <build> list`, which prints every cached query with its exact args to copy: one call instead of guessing. 3. MCP banners were not tee'd to TOOLCACHE_LOG, so shell and MCP hits could not be totalled from one file. Both now log there. Verification-run result for the record: 22 tool calls vs 81 on the same test cold (73% fewer), but the saving came overwhelmingly from the shared evidence file, not the call-level caches (1 shell hit of 8; MCP roughly a wash after the failed probes). The caches only pay on repeated identical lookups, and that run's second turn demanded a code path nobody had walked — a low hit rate there is the correct outcome, not a cache failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/cached-mcp.mjs | 43 +++++++++++++++++++++++++++++++++++++++---- lib/csv-state.mjs | 5 +++++ 2 files changed, 44 insertions(+), 4 deletions(-) diff --git a/bin/cached-mcp.mjs b/bin/cached-mcp.mjs index 8a57a8e..a5074ee 100644 --- a/bin/cached-mcp.mjs +++ b/bin/cached-mcp.mjs @@ -26,16 +26,29 @@ // Prefer storing a DIGEST rather than a raw payload: the point is to spare the // next reader the raw rows, not to relay them. -import { readFileSync } from "node:fs"; +import { readFileSync, readdirSync, existsSync, appendFileSync } from "node:fs"; +import { join } from "node:path"; import { toolCacheDirFor, mcpCacheKey, cacheGet, cachePut, cacheStats, isCacheableMcp, } from "../lib/tool-cache.mjs"; +// Same TOOLCACHE_LOG tee as cached-exec, so shell and MCP hits can be counted +// from one file. Previously only shell banners were logged, which made a run's +// combined hit rate impossible to total. +const logPath = process.env.TOOLCACHE_LOG ?? ""; +function banner(line) { + console.error(line); + if (logPath) { + try { appendFileSync(logPath, line + "\n", { encoding: "utf8", mode: 0o600 }); } catch { /* never break the call */ } + } +} + const [, , buildId, verb, tool, argsJson, writerId] = process.argv; if (!buildId || !verb) { console.error("usage: cached-mcp.mjs <buildId> get <tool> '<argsJson>'"); console.error(" cached-mcp.mjs <buildId> put <tool> '<argsJson>' <writerId> # payload on stdin"); + console.error(" cached-mcp.mjs <buildId> list # what is cached, with exact args to copy"); console.error(" cached-mcp.mjs <buildId> stats"); process.exit(2); } @@ -47,6 +60,28 @@ if (verb === "stats") { process.exit(0); } +// `list` exists because a HIT requires reproducing the args EXACTLY, and +// canonicalization only normalizes key ORDER, not content. A coordinator that +// guesses the logql/window/limit triple misses — one real run burned four +// probe calls guessing, to save two. Listing what is actually cached turns +// that into a single call: read the available queries, then `get` the one you +// want with its args copied verbatim. +if (verb === "list") { + if (!existsSync(dir)) { console.log("(no cache yet)"); process.exit(0); } + let n = 0; + for (const f of readdirSync(dir).filter((x) => x.endsWith(".json"))) { + let e; try { e = JSON.parse(readFileSync(join(dir, f), "utf8")); } catch { continue; } + if (!/^mcp__/.test(e.command ?? "")) continue; // shell entries live here too + n++; + const sp = e.command.indexOf(" "); + console.log(`\n[${e.key}] ${e.command.slice(0, sp)} (by ${e.writerId ?? "?"}, ${e.bytes}B)`); + console.log(` args: ${e.command.slice(sp + 1)}`); + console.log(` digest: ${String(e.stdout).replace(/\s+/g, " ").slice(0, 150)}…`); + } + if (!n) console.log("(no MCP entries cached — the orchestrator should pre-seed Step 4's queries)"); + process.exit(0); +} + if (!tool || argsJson === undefined) { console.error("both <tool> and '<argsJson>' are required"); process.exit(2); @@ -70,10 +105,10 @@ const key = mcpCacheKey(tool, args); if (verb === "get") { const hit = cacheGet(dir, key); if (!hit) { - console.error(`[mcp-cache MISS ${key} ${tool}] — make the MCP call, then 'put' the digest`); + banner(`[mcp-cache MISS ${key} ${tool}] — make the MCP call, then 'put' the digest`); process.exit(1); } - console.error(`[mcp-cache HIT ${key} ${tool} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + banner(`[mcp-cache HIT ${key} ${tool} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); process.stdout.write(hit.stdout); process.exit(0); } @@ -90,7 +125,7 @@ if (verb === "put") { process.exit(2); } const rec = cachePut(dir, key, { command: `${tool} ${argsJson}`, writerId, stdout: payload }, Date.now()); - console.error(`[mcp-cache STORED ${key} ${tool} — ${rec.bytes}B]`); + banner(`[mcp-cache STORED ${key} ${tool} — ${rec.bytes}B]`); process.exit(0); } diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 8590a66..422448f 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -52,6 +52,11 @@ export const COLUMNS = [ "possible_fix", "related_prs", "coverage", + // Both are part of the RCA_OUTPUT contract but had no column, so `flip` + // silently discarded them — `view_rca` in particular is the dashboard link + // the whole run exists to produce. + "view_rca", + "turns_used", "confidence", "timestamp", ]; From 96815a94c81668b0ac1d2d6d368c5bf9b4699d9e Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:49:03 +0530 Subject: [PATCH 19/51] feat(rca): window-guard PR table; name wrong-arity flip() calls precisely MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both from a live run on a second build (ObservabilityApiLaneSuite, 189 failures across 9 clusters). - `evidence-show --prs [repo]` prints mergedAt | #num | title plus the build's started_at. That table is the highest-value falsification per byte: on this run it disqualified 11 of 22 candidate PRs on the window guard alone, before a single diff was fetched. Previously a coordinator had to pipe --repo's raw JSON through an ad-hoc node one-liner to build it. - `flip(testRunId, fields)` — dropping the leading csvPath — used to bind an object to testRunId, read a nonexistent CSV, and return a bare `false` that a caller mistook for success. It now says exactly what went wrong and shows the correct signature. Cache behaviour on this build, worth recording: the shell cache was COLD, yet the representative still logged 4 hits of 7 fetches — concurrently dispatched siblings populated it for each other, including the single most expensive call (`gh pr diff 16450`). And `cached-mcp list` alone avoided two grafana calls: the seeded digests were self-sufficient, so no `get` was needed at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/evidence-show.mjs | 26 ++++++++++++++++++++++++++ lib/csv-state.mjs | 13 +++++++++++++ 2 files changed, 39 insertions(+) diff --git a/bin/evidence-show.mjs b/bin/evidence-show.mjs index 37d7bc1..7aa5ab8 100644 --- a/bin/evidence-show.mjs +++ b/bin/evidence-show.mjs @@ -31,6 +31,32 @@ if (mode === "--repo") { process.exit(0); } +// `--prs` prints the one table that does the most falsification work per byte: +// mergedAt | #num | title. A coordinator compares mergedAt against the build's +// start_at and disqualifies everything merged after it — no diffs fetched. On +// one real run that removed 11 of 22 candidates before a single `gh pr view`, +// and getting there previously required piping --repo's raw JSON through an +// ad-hoc node one-liner. +if (mode === "--prs") { + const repos = arg ? [arg] : Object.keys(folded.github ?? {}); + for (const repo of repos) { + const e = folded.github?.[repo]; + if (!e) { console.log(`${repo}: (not in evidence file)`); continue; } + const prs = e.prsInWindow ?? []; + const trust = e.prsSearched === true || prs.length > 0 ? "" : " [LIST NOT TRUSTWORTHY — never searched]"; + console.log(`\n${repo} (${prs.length} PR(s))${trust}`); + for (const p of prs.sort((a, b) => String(a.mergedAt).localeCompare(String(b.mergedAt)))) { + console.log(` ${p.mergedAt ?? "?".padEnd(24)} ${String(p.pr).padEnd(7)} ${String(p.title ?? "").slice(0, 88)}`); + } + } + const w = folded.suspectWindow; + if (w?.startedAt) { + console.log(`\nbuild started_at: ${w.startedAt}`); + console.log(" → anything merged AFTER that could not have shipped in this build (window guard)."); + } + process.exit(0); +} + if (mode !== "--summary") { console.log(JSON.stringify(folded, null, 2)); process.exit(0); diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 422448f..ee970c5 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -250,6 +250,19 @@ const COLUMN_ALIASES = new Map([ ]); export function flip(csvPath, testRunId, fields, nowMs) { + // Arity guard. `flip` is positional with csvPath FIRST, and a caller that + // drops it — `flip(testRunId, fields)` — otherwise binds an object to + // testRunId, reads a nonexistent CSV, and gets a bare `false` that is easy + // to mistake for success. Name the mistake precisely instead. + if (typeof csvPath !== "string" || (testRunId !== null && typeof testRunId === "object")) { + console.warn( + "[csv-state] flip called with the wrong arguments. Signature is " + + "flip(csvPath, testRunId, fields, nowMs) — csvPath FIRST, e.g. " + + "flip(csvPathFor(buildId), '3904695279', { status: 'RESOLVED', ... }, Date.now()). " + + `Got csvPath=${JSON.stringify(csvPath)?.slice(0, 60)}, testRunId=${JSON.stringify(testRunId)?.slice(0, 60)}. Row NOT written.`, + ); + return false; + } // Enforce the contract: a flip must name a valid outcome. A partial flip with // a missing/non-terminal rca_done would otherwise clear the claim yet leave the // row `pending` — re-exposing it for a duplicate RCA that clobbers this result. From 99c1858b4ae19287391d0e3c4cba8d110450c759 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:51:49 +0530 Subject: [PATCH 20/51] fix(rca): pad the log window past finished_at; forward direction; prove absence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A live run on a second build found the pre-fetch structurally blind to its own build's cause. An upstream outage ran 06:12:00-06:15:51 while the build's finished_at was 06:12:21, so a sweep scoped strictly to started_at..finished_at saw 21 seconds of a 4-minute outage. Step 4 now pads to started_at-2m .. finished_at+10m and labels findings inside/outside the strict window — without licensing an arbitrary window, which is the opposite failure (matching a coincidence in unrelated traffic). Two query mechanics added, both of which cost real calls in that run: - `direction` defaults to newest-first, so a limited query returns the END of the range. Verifying this finding required direction:"forward" — my own backward queries kept returning traffic clustered at each window's tail, which looked like the gap was absent when it was simply out of view. - Absence needs a control: a zero-result query is indistinguishable from a bad selector. Prove the logger was alive in the same window with a query you expect to be non-empty before treating silence as evidence. Verification note on the finding itself: the gap is real (zero /ext/v1 06:12:00->06:15:51, resuming with a 500 on /ext/v1/badge/build). Two secondary details in the coordinator's report did not hold — traffic resumes at 06:15:51 not 06:16:30, and badge traffic was not absent until 06:48. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- skills/rca-build/SKILL.md | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index e366b34..27b856d 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -253,10 +253,29 @@ gathers it*. A repo the connector can't reach records `{gap: "<reason>"}` — never blocks the rest of the pre-fetch. 4. For each workload: run the connector skill's compulsory kubectl + - VictoriaLogs sweep **once**, scoped to the build's own failure window - (`started_at`..`finished_at`, not "now" — see the connector skill's window - guidance). Persist via `setLogsEvidence(path, workload, + VictoriaLogs sweep **once**, anchored to the build's own clock — never + "now". **PAD the window: `started_at − 2m` .. `finished_at + 10m`.** + `finished_at` is when the build was *marked* finished, which is not when + the failing behaviour stopped: on one real build an upstream outage began + at 06:12:00 and ran to 06:15:51, while `finished_at` was 06:12:21 — a + sweep scoped strictly to `started_at..finished_at` saw 21 seconds of a + 4-minute outage and would have missed the cause entirely. Label every + finding with whether it falls inside or outside the strict window so a + coordinator can weigh it; do NOT silently widen to an arbitrary window + (that is the separate, opposite failure of matching a coincidence from + unrelated traffic). Persist via `setLogsEvidence(path, workload, {clusterIds, kubectlSweep, victorialogs, gap}, nowMs)`. + + Two query mechanics that cost real calls when missed: + - **`direction` defaults to newest-first**, so a limited query always + returns the END of the window. To find when something *started* — the + first request after a gap, the onset of an error burst — pass + `direction: "forward"`. A gap "confirmed" from a backward query is not + confirmed at all; it is just the tail of the range. + - **Absence needs a control.** A zero-result query is indistinguishable + from a wrong selector. Before reporting "no traffic", prove the logger + was alive in the same window with a query you expect to be non-empty + (e.g. readiness probes from a named pod). Only then is silence evidence. 5. `resolveBaseline(lastGreenRef, fallbackRef)` (from `lib/evidence-cache.mjs`) → `setBaseline(path, baseline, suspectWindow, nowMs)`. No "last green" baseline (never-green suite) → fall back to a configured baseline ref and From 94cad2d75fa66adc0e0a9bf07483bf56eb005561 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Fri, 31 Jul 2026 20:57:57 +0530 Subject: [PATCH 21/51] docs(rca): distinguish the two TFA failures; size-check before trusting a negative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both from a live run on a second build. - `TFA agent run failed` (the wedge, size-independent, fix = resubmit on the same thread) and `turn expired or not found` (observed on a ~2000-char over-cap submit) are different failures. The latter's text names a thread/turn problem, so a coordinator reads it as a wedge and resubmits unchanged instead of shortening. Named both explicitly. - A coordinator nearly concluded a manifest lacked an entry when the fetch had been cut at ~64KB; its own `wc -l` control caught it (1042 lines vs 1518). Investigated and the tool cache is NOT the cause — verified 104KB and 214KB files round-trip intact, and the only truncation is past 256KB with an explicit marker — so this was surrounding tool plumbing. The hazard is real regardless of layer, and a silent truncation converts "grep found nothing" into a false negative, so: size-check a large fetch before treating an absent match as evidence of absence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index afe92c4..6c3deec 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -174,6 +174,24 @@ read-only and has no side effects, so a read is always safe to repeat. NOT to mint a new thread and not to end the run `PENDING`. Ending PENDING here throws away a resolvable test. Only stop once the turn cap is spent. +4b-i. **Two DIFFERENT TFA failures, don't confuse them.** + - `TFA agent run failed` — the wedge. Unrelated to message size (a + 240-char message wedged like a 1500-char one). Fix: resubmit on the same + thread, per 4b. + - `turn expired or not found` — observed on an over-cap (~2000-char) + submit. The text names a thread/turn problem, which reads as a wedge and + sends you down the wrong path; it is really a size rejection. If you see + this, shorten and resend before assuming the thread is broken. + +4b-ii. **Size-check any large fetch before trusting a negative result.** A + truncated payload turns "grep found nothing" into a false negative, and it + is silent. A coordinator nearly concluded a manifest didn't contain an + entry when the file had simply been cut at ~64KB — its own `wc -l` check + is what caught it (1042 lines vs 1518 real). The tool cache does not do + this (it truncates only past 256KB, and marks it), but the surrounding + tool plumbing can. So on any fetch of a big file: verify size or line + count first, and only then treat an absent match as evidence of absence. + 4c. **Keep every turn message under `turnMessageMaxChars` (1000)** — for digest discipline, NOT as a wedge cure. An early correlation suggested oversized messages caused the turn wedge (~1400/~1350-char submits failed From 20b8bf778006a34d745681d87e301b845614ffc5 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 13:56:57 +0530 Subject: [PATCH 22/51] perf(rca): pre-fetch PR changed-file paths in the existing PR-list call MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured across three real runs (407 gh calls): per-PR `gh pr view <n> --json files` was 44 calls (10.8%), and every one is avoidable. `gh pr list` accepts `--json files`, so ONE request per repo returns every PR's changed paths — the Step 4 call we already make, just asking for one more field. Verified live on build fiehz…: 2 calls covered 22 PRs / 104 changed paths, replacing up to 22 per-PR fetches across coordinators. Path-overlap — the first falsification test in github-evidence.md — is now answerable from the evidence file with zero calls. It independently surfaces testhub#16450 -> project.js, the culprit a coordinator previously spent calls hunting. Two other measured wastes named in the same step: connector re-probes (`gh auth status` / `kubectl version`) were 17 calls (4.2%) purely because the manifest wasn't trusted, and file CONTENTS were 31% but only partly predictable — so those are deliberately NOT bulk-fetched. The `files` lists tell a coordinator which files matter; the tool cache dedupes the ones two coordinators both open. Diffs stay on-demand: large, and few PRs need one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- skills/rca-build/SKILL.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 27b856d..3216e88 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -252,6 +252,35 @@ gathers it*. `setGithubEvidence(path, repo, {deployState, prsInWindow, gap}, nowMs)`. A repo the connector can't reach records `{gap: "<reason>"}` — never blocks the rest of the pre-fetch. + + **Ask for `files` in the PR-list call — it costs nothing extra and is the + single highest-leverage thing in this step:** + + ```bash + gh pr list -R <org>/<repo> --state merged --base <branch> \ + --search 'merged:<from>..<to>' --json number,title,mergedAt,url,files --limit 100 + ``` + + `--json files` returns every PR's changed paths in the SAME call, so one + request per repo replaces one `gh pr view <n> --json files` per PR across + every coordinator. Measured across three real runs, per-PR file-list + fetches were **44 of 407 gh calls (10.8%)** — all of them avoidable here. + Store the paths in each PR's `files` field rather than leaving it `null`: + path-overlap is the first falsification test in + `references/github-evidence.md`, so with `files` populated a coordinator + rules a suspect in or out from the evidence file alone, and only fetches a + diff for the handful that survive. Do NOT pre-fetch diffs — those are large + and only a few PRs ever need one. + + Two other measured wastes this step should pre-empt: + - **Never let coordinators re-probe connectors.** `gh auth status` / + `kubectl version` accounted for **17 of 407 gh calls (4.2%)** purely + because the manifest wasn't trusted. State plainly in the dispatch prompt + that the gate validated them. + - **File contents were 31% of gh traffic** and are only partly predictable, + so do NOT bulk-fetch them. The `files` lists above tell a coordinator + exactly which files matter, and the tool cache dedupes the ones two + coordinators both open. 4. For each workload: run the connector skill's compulsory kubectl + VictoriaLogs sweep **once**, anchored to the build's own clock — never "now". **PAD the window: `started_at − 2m` .. `finished_at + 10m`.** From 19e0f5fc3c690ea17f9c30be8eff2c3c02253f63 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:04:14 +0530 Subject: [PATCH 23/51] perf(rca): read repo files from local clones, sha-pinned, instead of gh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit File CONTENTS were 126 of 407 gh calls (31%) across three real runs — the largest remaining slice. Six of the eight repos this workspace needs are already cloned locally, so most of those round-trips are avoidable. Measured: `gh api .../contents/<path>?ref=<sha>` ~1022ms vs `git show <sha>:<path>` ~37ms — 27x faster, byte-identical. A stale clone needs one targeted `git fetch` (~5.4s), which pays for itself after ~6 reads of that repo. SHA-PINNED ONLY, and this is the whole reason the module is careful rather than a one-liner. This workspace's clones were 12 commits behind: reading testPlan.js from the local BRANCH returned 281,061 bytes where the real head had 282,315. RCA reasons about what changed in a window, so silently reading different code produces a confident wrong answer. Pinned to the build-time sha — which Step 4 already records as deployState — content is byte-identical and staleness stops mattering, because a commit either exists locally or it provably doesn't. Two deliberate non-behaviours: the library never falls back to the network itself (it reports remote-needed and lets the edge decide, so a local answer can't be silently substituted), and a path genuinely absent at that commit is returned as an ANSWER rather than a fallback trigger. Remote reads still go through the tool cache, so a repo without a local clone behaves exactly as before. 137 tests pass, including a real two-commit git fixture that exercises the stale-branch-vs-pinned-sha distinction rather than mocking it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/repo-read.mjs | 76 +++++++++++++++++++++++++++ lib/repo-source.mjs | 102 +++++++++++++++++++++++++++++++++++++ tests/repo-source.test.mjs | 82 +++++++++++++++++++++++++++++ 3 files changed, 260 insertions(+) create mode 100644 bin/repo-read.mjs create mode 100644 lib/repo-source.mjs create mode 100644 tests/repo-source.test.mjs diff --git a/bin/repo-read.mjs b/bin/repo-read.mjs new file mode 100644 index 0000000..4128d62 --- /dev/null +++ b/bin/repo-read.mjs @@ -0,0 +1,76 @@ +#!/usr/bin/env node +// Read a repo file at a pinned commit, preferring a local clone over the +// network, and falling back to `gh` automatically. +// +// node bin/repo-read.mjs <buildId> <writerId> <org/repo> <sha> <path> [--fetch] +// +// Measured on this workspace: local `git show` ~37ms vs `gh api` ~1022ms for +// the same file, byte-identical. One targeted `git fetch` (~5s) makes a stale +// clone usable, so the fetch pays for itself after ~6 reads of that repo. +// +// SHA-PINNED ONLY. A branch name is refused: local clones are routinely behind +// (12 commits, on this machine), and reading a branch locally returned +// different bytes than the real head — which for RCA means confidently +// reasoning about code that never shipped. Get the sha from the evidence +// file's `deployState` (branch tip at build start), which Step 4 records. +// +// Remote results still go through the tool cache, so a repo with no local +// clone degrades to exactly the previous behaviour. + +import { execFileSync } from "node:child_process"; +import { readFileAt } from "../lib/repo-source.mjs"; +import { toolCacheDirFor, cacheKey, cacheGet, cachePut } from "../lib/tool-cache.mjs"; + +const [, , buildId, writerId, repo, sha, path, ...flags] = process.argv; +if (!buildId || !writerId || !repo || !sha || !path) { + console.error("usage: repo-read.mjs <buildId> <writerId> <org/repo> <sha> <path> [--fetch]"); + console.error(" sha must be a COMMIT SHA (a branch name is refused — it can read stale code)"); + process.exit(2); +} + +const workspaceRoot = process.env.RCA_WORKSPACE_ROOT ?? "/Users/harshitm/Desktop/browserstack"; +const branch = process.env.RCA_SHIPPING_BRANCH ?? "observability_pre_prod"; +const allowFetch = flags.includes("--fetch"); + +const local = readFileAt({ repo, sha, path, workspaceRoot, branch, allowFetch }); + +if (local.ok) { + console.error(`[repo-read LOCAL ${repo}@${sha.slice(0, 8)} ${local.content.length}B — no network]`); + process.stdout.write(local.content); + process.exit(0); +} + +// A path genuinely absent at that commit is an answer; don't re-ask the network. +if (local.source === "local") { + console.error(`[repo-read LOCAL ${repo}@${sha.slice(0, 8)}] ${local.reason}`); + process.exit(1); +} + +console.error(`[repo-read -> remote] ${local.reason}`); + +const dir = toolCacheDirFor(buildId, process.env.RCA_STATE_DIR ?? ""); +const cmd = `gh api repos/${repo}/contents/${path}?ref=${sha}`; +const key = cacheKey(cmd); +const hit = cacheGet(dir, key); +if (hit) { + console.error(`[repo-read CACHE HIT ${key} — captured by ${hit.writerId ?? "?"}, ${hit.bytes}B]`); + process.stdout.write(hit.stdout); + process.exit(0); +} + +let out; +try { + const raw = execFileSync("gh", ["api", `repos/${repo}/contents/${path}?ref=${sha}`, "--jq", ".content"], + { encoding: "utf8", maxBuffer: 64 * 1024 * 1024, stdio: ["ignore", "pipe", "pipe"] }); + out = Buffer.from(raw.replace(/\s+/g, ""), "base64").toString("utf8"); +} catch (err) { + if (err.stderr) process.stderr.write(err.stderr.toString()); + console.error(`[repo-read REMOTE failed, NOT cached]`); + process.exit(typeof err.status === "number" ? err.status : 1); +} + +if (out.trim() !== "") { + cachePut(dir, key, { command: cmd, writerId, stdout: out, exitCode: 0 }, Date.now()); + console.error(`[repo-read REMOTE ${out.length}B — cached]`); +} +process.stdout.write(out); diff --git a/lib/repo-source.mjs b/lib/repo-source.mjs new file mode 100644 index 0000000..cb35539 --- /dev/null +++ b/lib/repo-source.mjs @@ -0,0 +1,102 @@ +// Read repo files from a LOCAL clone when one is available, instead of paying +// a network round-trip per file. +// +// Measured: `gh api .../contents/<path>?ref=<sha>` ~1022ms; the same read as +// `git show <sha>:<path>` from a local clone ~37ms — 27x faster, and +// byte-identical. Across three real runs, file CONTENTS were 126 of 407 gh +// calls (31%) and commit history another 48 (12%), so this is the largest +// remaining slice of github traffic. +// +// THE CORRECTNESS RULE: ALWAYS PIN TO A COMMIT SHA, NEVER A BRANCH NAME. +// +// This is not pedantry — it is the whole reason this module needs care. A +// developer's clone is usually stale: measured on this workspace, +// `origin/observability_pre_prod` was 12 commits behind, and reading +// `testPlan.js` from the local branch returned 281,061 bytes where the real +// branch head had 282,315. For RCA that is catastrophic in a quiet way: we +// reason about *what changed in a window*, so silently reading different code +// yields a confident wrong answer. Pinned to the build-time SHA the content is +// byte-identical to GitHub, and staleness stops mattering — a commit either +// exists locally or it does not, and we can tell which. + +import { execFileSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; + +const SHA = /^[0-9a-f]{7,40}$/i; + +function git(dir, args) { + return execFileSync("git", ["-C", dir, ...args], { + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }); +} + +/** Local clone path for `org/repo`, if one exists under `workspaceRoot`. + * Matches on the bare repo name, which is how these workspaces are laid out. */ +export function localCloneFor(repo, workspaceRoot) { + const name = String(repo).split("/").pop(); + const dir = join(workspaceRoot, name); + return existsSync(join(dir, ".git")) ? dir : null; +} + +/** Is this exact commit present locally? The only question that matters — + * a present commit is immutable, so its content cannot be stale. */ +export function hasCommit(dir, sha) { + try { + git(dir, ["cat-file", "-e", `${sha}^{commit}`]); + return true; + } catch { + return false; + } +} + +/** Make `sha` available locally with one targeted fetch. Returns true if the + * commit is present afterwards. Fetch touches only remote-tracking refs — it + * never moves a branch or the working tree, so it is safe to run against a + * repo someone is working in. */ +export function ensureCommit(dir, sha, branch) { + if (hasCommit(dir, sha)) return true; + try { + git(dir, ["fetch", "--quiet", "origin", branch ?? sha]); + } catch { + return false; + } + return hasCommit(dir, sha); +} + +/** + * Read one file at one commit. Returns + * `{ ok, content, source: "local"|"remote-needed", reason }`. + * + * Deliberately does NOT fall back to the network itself — it reports that the + * caller should. Keeping the decision at the edge means a wrong-looking local + * answer can never be silently substituted for the real one. + */ +export function readFileAt({ repo, sha, path, workspaceRoot, branch, allowFetch = false }) { + if (!SHA.test(String(sha ?? ""))) { + // Refusing a branch name is the point — see the header. + return { ok: false, source: "remote-needed", reason: `ref must be a commit sha, got ${JSON.stringify(sha)} (a branch name can silently read stale code)` }; + } + const dir = localCloneFor(repo, workspaceRoot); + if (!dir) return { ok: false, source: "remote-needed", reason: `no local clone of ${repo} under ${workspaceRoot}` }; + + if (!hasCommit(dir, sha)) { + if (!allowFetch) return { ok: false, source: "remote-needed", reason: `commit ${sha} not present in ${dir} (pass allowFetch to fetch it once)` }; + if (!ensureCommit(dir, sha, branch)) { + return { ok: false, source: "remote-needed", reason: `commit ${sha} still absent after fetch` }; + } + } + try { + return { ok: true, source: "local", content: git(dir, ["show", `${sha}:${path}`]), dir }; + } catch (err) { + // A missing path at that commit is a real answer, not a fallback trigger: + // the file genuinely did not exist there. + const msg = String(err.stderr ?? err.message ?? ""); + if (/does not exist|exists on disk, but not in/i.test(msg)) { + return { ok: false, source: "local", reason: `path not present at ${sha}: ${path}` }; + } + return { ok: false, source: "remote-needed", reason: msg.slice(0, 200) }; + } +} diff --git a/tests/repo-source.test.mjs b/tests/repo-source.test.mjs new file mode 100644 index 0000000..cfc62cb --- /dev/null +++ b/tests/repo-source.test.mjs @@ -0,0 +1,82 @@ +import { test, beforeEach, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { localCloneFor, hasCommit, readFileAt } from "../lib/repo-source.mjs"; + +let ws, repoDir, sha1, sha2; + +// Build a real throwaway git repo with two commits, so the staleness scenario +// is exercised for real rather than mocked. +beforeEach(() => { + ws = mkdtempSync(join(tmpdir(), "rca-ws-")); + repoDir = join(ws, "testrepo"); + mkdirSync(repoDir); + const g = (...a) => execFileSync("git", ["-C", repoDir, ...a], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }); + g("init", "-q"); + g("config", "user.email", "t@t.t"); + g("config", "user.name", "t"); + writeFileSync(join(repoDir, "app.js"), "VERSION_ONE\n"); + g("add", "."); g("commit", "-qm", "one"); + sha1 = g("rev-parse", "HEAD").trim(); + writeFileSync(join(repoDir, "app.js"), "VERSION_TWO\n"); + g("add", "."); g("commit", "-qm", "two"); + sha2 = g("rev-parse", "HEAD").trim(); +}); +afterEach(() => rmSync(ws, { recursive: true, force: true })); + +test("localCloneFor finds a clone by bare repo name", () => { + assert.equal(localCloneFor("browserstack/testrepo", ws), repoDir); + assert.equal(localCloneFor("browserstack/not-cloned", ws), null); +}); + +test("hasCommit distinguishes present from absent commits", () => { + assert.equal(hasCommit(repoDir, sha1), true); + assert.equal(hasCommit(repoDir, "0".repeat(40)), false); +}); + +// The core safety property. A branch name resolves to whatever the clone +// happens to have, which on a real machine was 12 commits stale and returned +// different bytes than the true head — a silently wrong RCA input. +test("a branch name is REFUSED; only a commit sha is accepted", () => { + const r = readFileAt({ repo: "browserstack/testrepo", sha: "main", path: "app.js", workspaceRoot: ws }); + assert.equal(r.ok, false); + assert.equal(r.source, "remote-needed"); + assert.match(r.reason, /must be a commit sha/); +}); + +test("a pinned sha reads the content AT THAT COMMIT, not the tip", () => { + const older = readFileAt({ repo: "browserstack/testrepo", sha: sha1, path: "app.js", workspaceRoot: ws }); + assert.equal(older.ok, true); + assert.equal(older.source, "local"); + assert.equal(older.content.trim(), "VERSION_ONE", "must read the old commit, not HEAD"); + + const newer = readFileAt({ repo: "browserstack/testrepo", sha: sha2, path: "app.js", workspaceRoot: ws }); + assert.equal(newer.content.trim(), "VERSION_TWO"); +}); + +test("no local clone -> defers to the caller for a remote read", () => { + const r = readFileAt({ repo: "browserstack/absent", sha: sha1, path: "app.js", workspaceRoot: ws }); + assert.equal(r.ok, false); + assert.equal(r.source, "remote-needed"); + assert.match(r.reason, /no local clone/); +}); + +test("commit absent locally -> remote-needed, and does NOT fetch unless asked", () => { + const r = readFileAt({ repo: "browserstack/testrepo", sha: "0".repeat(40), path: "app.js", workspaceRoot: ws }); + assert.equal(r.ok, false); + assert.equal(r.source, "remote-needed"); + assert.match(r.reason, /not present|allowFetch/); +}); + +// A path that genuinely didn't exist at that commit is an ANSWER. Treating it +// as a fallback trigger would send the caller to the network to be told the +// same thing, and risks a tip-of-branch read papering over the real history. +test("path missing at that commit is a local answer, not a remote fallback", () => { + const r = readFileAt({ repo: "browserstack/testrepo", sha: sha1, path: "nope.js", workspaceRoot: ws }); + assert.equal(r.ok, false); + assert.equal(r.source, "local"); + assert.match(r.reason, /path not present/); +}); From aa00966345ed98510f23b7c19182b8d841edb402 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:18:01 +0530 Subject: [PATCH 24/51] =?UTF-8?q?fix(rca):=20keep=20the=20local-repo=20rea?= =?UTF-8?q?der=20generic=20=E2=80=94=20no=20product=20or=20machine=20liter?= =?UTF-8?q?als?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first cut of bin/repo-read.mjs defaulted RCA_WORKSPACE_ROOT to a specific developer's path and RCA_SHIPPING_BRANCH to observability_pre_prod. That is the exact coupling the capability-manifest design exists to prevent: which repos exist, where they are checked out, and what branch ships are facts the product CONNECTOR SKILL owns and the gate resolves, not something the plugin may assume. As written it would have worked for one product on one machine and silently misbehaved elsewhere. Both are now required inputs. Missing RCA_WORKSPACE_ROOT fails loudly with an explanation instead of guessing; the shipping branch is optional and only widens a fetch on a miss. lib/repo-source.mjs names no repo, branch or path — callers pass everything. Audited lib/, bin/, workflows/ and config/: zero product literals remain. 137 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/repo-read.mjs | 23 +++++++++++++++++++++-- lib/repo-source.mjs | 2 +- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/bin/repo-read.mjs b/bin/repo-read.mjs index 4128d62..3a6adc9 100644 --- a/bin/repo-read.mjs +++ b/bin/repo-read.mjs @@ -14,6 +14,10 @@ // reasoning about code that never shipped. Get the sha from the evidence // file's `deployState` (branch tip at build start), which Step 4 records. // +// Generic by construction: the workspace root and shipping branch are INPUTS +// (RCA_WORKSPACE_ROOT / RCA_SHIPPING_BRANCH) supplied by the gate and the +// product's connector skill. This file names no repo, no branch and no path. +// // Remote results still go through the tool cache, so a repo with no local // clone degrades to exactly the previous behaviour. @@ -28,8 +32,23 @@ if (!buildId || !writerId || !repo || !sha || !path) { process.exit(2); } -const workspaceRoot = process.env.RCA_WORKSPACE_ROOT ?? "/Users/harshitm/Desktop/browserstack"; -const branch = process.env.RCA_SHIPPING_BRANCH ?? "observability_pre_prod"; +// NO DEFAULTS for either of these. The plugin is generic over product and +// infra: which repos exist, where they are checked out, and what branch ships +// are facts the CONNECTOR SKILL owns and the gate resolves — never something +// this plugin should assume. Baking in a workspace path or a branch name would +// silently make the plugin work for exactly one product on exactly one +// machine, which is the failure mode the whole capability-manifest design +// exists to avoid. +const workspaceRoot = process.env.RCA_WORKSPACE_ROOT; +if (!workspaceRoot) { + console.error("[repo-read] RCA_WORKSPACE_ROOT is not set."); + console.error(" It must come from the gate/connector skill — the plugin does not assume a workspace layout."); + console.error(" Set it to the directory holding the local clones, e.g. RCA_WORKSPACE_ROOT=$(pwd)/.."); + process.exit(2); +} +// Only needed to widen a fetch on a miss; a sha-only fetch is attempted when +// absent. Supplied by the connector skill, which knows the shipping branch. +const branch = process.env.RCA_SHIPPING_BRANCH || undefined; const allowFetch = flags.includes("--fetch"); const local = readFileAt({ repo, sha, path, workspaceRoot, branch, allowFetch }); diff --git a/lib/repo-source.mjs b/lib/repo-source.mjs index cb35539..be96e24 100644 --- a/lib/repo-source.mjs +++ b/lib/repo-source.mjs @@ -11,7 +11,7 @@ // // This is not pedantry — it is the whole reason this module needs care. A // developer's clone is usually stale: measured on this workspace, -// `origin/observability_pre_prod` was 12 commits behind, and reading +// the shipping branch's remote-tracking ref was 12 commits behind, and reading // `testPlan.js` from the local branch returned 281,061 bytes where the real // branch head had 282,315. For RCA that is catastrophic in a quiet way: we // reason about *what changed in a window*, so silently reading different code From 286884e4d5be745bf4154cb62e09a7924f231dc9 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:22:42 +0530 Subject: [PATCH 25/51] feat(rca): bounded workspace discovery, resolved once and shared via evidence file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-ups to the local-repo reader. GENERIC DISCOVERY. discoverWorkspaceRoot() finds the clone directory without any hardcoded path: it tries the caller's dir, its parent, then grandparent, and accepts one only if it actually contains a repo from THIS run's validated list. That check is what makes it generic — a different product with a different layout resolves by the same rule, and nothing about repos, branches or paths is assumed. It is deliberately bounded to ~3 tries: climbing further risks matching an unrelated checkout, which is silently wrong rather than merely slow, and "not found" is a perfectly good answer since the caller falls back to the network. An explicit RCA_WORKSPACE_ROOT is honoured but still verified, so a stale override fails loudly instead of quietly. RESOLVED ONCE, NOT PER-AGENT. resolveLocalRepos() + setLocalRepos() record, in the evidence file, which repos are readable locally at their pinned shas. A coordinator reads that map and knows immediately whether to go local or remote, with no filesystem probing of its own — the same duplication the evidence file already removes for PRs and logs. Resolution order in bin/repo-read.mjs is now: evidence file, then RCA_WORKSPACE_ROOT, then bounded discovery, then network. Verified end to end on this workspace: gate discovered the root in 2 tries, resolved 3 of 4 repos as locally readable (the fourth has no clone and falls through to the cached gh path), and a coordinator then read a 145KB file with no network and no probing, sourced "via evidence-file (resolved at gate)". 144 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- bin/repo-read.mjs | 50 +++++++++++++++++++-------- lib/evidence-file.mjs | 23 ++++++++++++ lib/repo-source.mjs | 71 ++++++++++++++++++++++++++++++++++++++ tests/repo-source.test.mjs | 68 +++++++++++++++++++++++++++++++++++- 4 files changed, 196 insertions(+), 16 deletions(-) diff --git a/bin/repo-read.mjs b/bin/repo-read.mjs index 3a6adc9..dc79169 100644 --- a/bin/repo-read.mjs +++ b/bin/repo-read.mjs @@ -22,7 +22,7 @@ // clone degrades to exactly the previous behaviour. import { execFileSync } from "node:child_process"; -import { readFileAt } from "../lib/repo-source.mjs"; +import { readFileAt, discoverWorkspaceRoot } from "../lib/repo-source.mjs"; import { toolCacheDirFor, cacheKey, cacheGet, cachePut } from "../lib/tool-cache.mjs"; const [, , buildId, writerId, repo, sha, path, ...flags] = process.argv; @@ -32,29 +32,49 @@ if (!buildId || !writerId || !repo || !sha || !path) { process.exit(2); } -// NO DEFAULTS for either of these. The plugin is generic over product and -// infra: which repos exist, where they are checked out, and what branch ships -// are facts the CONNECTOR SKILL owns and the gate resolves — never something -// this plugin should assume. Baking in a workspace path or a branch name would -// silently make the plugin work for exactly one product on exactly one -// machine, which is the failure mode the whole capability-manifest design -// exists to avoid. -const workspaceRoot = process.env.RCA_WORKSPACE_ROOT; +// NO HARDCODED WORKSPACE OR BRANCH. Which repos exist, where they are checked +// out, and what branch ships are facts the CONNECTOR SKILL owns and the gate +// resolves. Baking either in would make the plugin work for exactly one +// product on one machine — the coupling the capability-manifest design exists +// to avoid. +// +// Resolution order, cheapest and most authoritative first: +// 1. the evidence file's `localRepos` — resolved ONCE at the gate, so a +// coordinator does no filesystem probing at all; +// 2. RCA_WORKSPACE_ROOT, if the caller set it; +// 3. a bounded structural guess (this dir, its parent, grandparent), +// accepted only if it actually contains the repo being asked for. +// Anything else: give up and use the network. Guessing harder risks reading +// an unrelated checkout, which is silently wrong rather than merely slow. +let workspaceRoot = process.env.RCA_WORKSPACE_ROOT; +let rootSource = workspaceRoot ? "RCA_WORKSPACE_ROOT" : null; + +const evidencePath = process.env.RCA_EVIDENCE_FILE; +if (!workspaceRoot && evidencePath) { + try { + const { readEvidenceFile } = await import("../lib/evidence-file.mjs"); + const lr = readEvidenceFile(evidencePath)?.localRepos; + if (lr?.workspaceRoot) { workspaceRoot = lr.workspaceRoot; rootSource = "evidence-file (resolved at gate)"; } + } catch { /* evidence file optional */ } +} + if (!workspaceRoot) { - console.error("[repo-read] RCA_WORKSPACE_ROOT is not set."); - console.error(" It must come from the gate/connector skill — the plugin does not assume a workspace layout."); - console.error(" Set it to the directory holding the local clones, e.g. RCA_WORKSPACE_ROOT=$(pwd)/.."); - process.exit(2); + const here = new URL("..", import.meta.url).pathname; + const d = discoverWorkspaceRoot({ repos: [repo], from: here, maxTries: 3 }); + if (d.root) { workspaceRoot = d.root; rootSource = `auto-discovered (${d.tried.length} tr${d.tried.length === 1 ? "y" : "ies"})`; } + else console.error(`[repo-read] no local workspace found — ${d.reason}`); } // Only needed to widen a fetch on a miss; a sha-only fetch is attempted when // absent. Supplied by the connector skill, which knows the shipping branch. const branch = process.env.RCA_SHIPPING_BRANCH || undefined; const allowFetch = flags.includes("--fetch"); -const local = readFileAt({ repo, sha, path, workspaceRoot, branch, allowFetch }); +const local = workspaceRoot + ? readFileAt({ repo, sha, path, workspaceRoot, branch, allowFetch }) + : { ok: false, source: "remote-needed", reason: "no local workspace resolved" }; if (local.ok) { - console.error(`[repo-read LOCAL ${repo}@${sha.slice(0, 8)} ${local.content.length}B — no network]`); + console.error(`[repo-read LOCAL ${repo}@${sha.slice(0, 8)} ${local.content.length}B — no network, root via ${rootSource}]`); process.stdout.write(local.content); process.exit(0); } diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index 2e76215..604916f 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -112,6 +112,11 @@ export function emptyEvidenceFile(buildId, nowMs) { suspectWindow: null, github: {}, logs: {}, + // Where each repo can be read locally at its pinned sha, resolved ONCE at + // the gate. Without this every coordinator re-probes the filesystem for + // the workspace and re-checks each commit — pure duplicated setup, which + // is the same waste the evidence file exists to remove for PRs and logs. + localRepos: null, coverage: { reposCovered: [], reposGapped: [], workloadsCovered: [], workloadsGapped: [] }, }; } @@ -224,6 +229,24 @@ export function initEvidenceFile(filePath, buildId, nowMs) { return doc; } +/** + * Persist the once-resolved local-repo map (from `repo-source.mjs`'s + * `discoverWorkspaceRoot` + `resolveLocalRepos`). Shape: + * `{ workspaceRoot, repos: { "org/repo": {dir, sha, usable, reason} } }`. + * + * A coordinator reads this and immediately knows, per repo, whether to use a + * local sha-pinned read or go to the network — with no filesystem probing of + * its own. `workspaceRoot: null` is a legitimate, useful answer: it means + * discovery ran and failed, so nobody should try again. + */ +export function setLocalRepos(filePath, localRepos, nowMs) { + const doc = loadOrInit(filePath, nowMs); + doc.localRepos = localRepos; + doc.generatedAtMs = nowMs; + writeEvidenceFile(filePath, doc); + return doc.localRepos; +} + /** Records the diff/PR-window baseline once, at the start of the Step 4 pass. * `baseline` is `resolveBaseline(...)`'s return value from `evidence-cache.mjs` * (`{ref, isFallback}`); `suspectWindow` is whatever shape the active connector diff --git a/lib/repo-source.mjs b/lib/repo-source.mjs index be96e24..865430e 100644 --- a/lib/repo-source.mjs +++ b/lib/repo-source.mjs @@ -41,6 +41,77 @@ export function localCloneFor(repo, workspaceRoot) { return existsSync(join(dir, ".git")) ? dir : null; } +/** + * Find the directory holding the local clones, WITHOUT hardcoding a path. + * + * `repos` is the gate's validated repo list, and it is what makes this + * generic: a candidate only wins if it actually contains one of the repos + * THIS run cares about. No repo name, product or path is baked in — a + * different product with a different checkout layout resolves by the same + * rule. + * + * Bounded on purpose: an explicit override, then at most `maxTries` + * structural guesses. Searching the filesystem for a plausible-looking + * directory would risk picking a stale or unrelated checkout, and a wrong + * workspace silently yields the wrong source code — the same class of failure + * as reading a stale branch. Finding nothing is a fine answer; the caller + * falls back to the network. + * + * Returns `{ root, matched, tried }`, or `{ root: null, tried }`. + */ +export function discoverWorkspaceRoot({ repos = [], explicit, from, maxTries = 3 } = {}) { + const verify = (dir) => { + if (!dir || !existsSync(dir)) return null; + const hit = repos.find((r) => localCloneFor(r, dir)); + return hit ? { root: dir, matched: hit } : null; + }; + + // An explicit value is authoritative and not counted as a guess — but it is + // still verified, so a stale env var fails loudly instead of quietly. + if (explicit) { + const ok = verify(explicit); + return ok ? { ...ok, tried: [explicit] } : { root: null, tried: [explicit], reason: `RCA_WORKSPACE_ROOT=${explicit} contains none of the validated repos` }; + } + + // Structural guesses only, in decreasing confidence. `from` is typically the + // plugin's own directory, which usually sits inside the workspace. + const base = from ?? process.cwd(); + const candidates = [base, join(base, ".."), join(base, "..", "..")].slice(0, maxTries); + + const tried = []; + for (const c of candidates) { + tried.push(c); + const ok = verify(c); + if (ok) return { ...ok, tried }; + } + return { root: null, tried, reason: `none of ${tried.length} candidate(s) contained any of: ${repos.join(", ") || "(no repos given)"}` }; +} + +/** + * Resolve, once, which of `repos` are readable locally at their pinned shas. + * The result is meant to be persisted (evidence file) so that every later + * coordinator reads a map instead of re-probing the filesystem. + * + * `pins` is `{ "org/repo": "<sha>" }` — normally the deployState shas Step 4 + * already computed. + */ +export function resolveLocalRepos({ repos, pins = {}, workspaceRoot, branch, allowFetch = false }) { + const out = {}; + for (const repo of repos) { + const dir = localCloneFor(repo, workspaceRoot); + if (!dir) { out[repo] = { dir: null, usable: false, reason: "no local clone" }; continue; } + const sha = pins[repo]; + if (!sha) { out[repo] = { dir, usable: false, reason: "no pinned sha for this repo" }; continue; } + let present = hasCommit(dir, sha); + let fetched = false; + if (!present && allowFetch) { fetched = ensureCommit(dir, sha, branch); present = fetched; } + out[repo] = present + ? { dir, sha, usable: true, fetched } + : { dir, sha, usable: false, reason: `commit ${sha} not present locally${allowFetch ? " even after fetch" : ""}` }; + } + return out; +} + /** Is this exact commit present locally? The only question that matters — * a present commit is immutable, so its content cannot be stale. */ export function hasCommit(dir, sha) { diff --git a/tests/repo-source.test.mjs b/tests/repo-source.test.mjs index cfc62cb..08cfb40 100644 --- a/tests/repo-source.test.mjs +++ b/tests/repo-source.test.mjs @@ -4,7 +4,7 @@ import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs"; import { execFileSync } from "node:child_process"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { localCloneFor, hasCommit, readFileAt } from "../lib/repo-source.mjs"; +import { localCloneFor, hasCommit, readFileAt, discoverWorkspaceRoot, resolveLocalRepos } from "../lib/repo-source.mjs"; let ws, repoDir, sha1, sha2; @@ -74,6 +74,72 @@ test("commit absent locally -> remote-needed, and does NOT fetch unless asked", // A path that genuinely didn't exist at that commit is an ANSWER. Treating it // as a fallback trigger would send the caller to the network to be told the // same thing, and risks a tip-of-branch read papering over the real history. +// The real shape: the plugin lives one level inside the workspace, alongside +// the clones, so the root is found on the second try. +test("discoverWorkspaceRoot walks up to the dir holding the clones", () => { + const pluginDir = join(ws, "some-plugin"); + mkdirSync(pluginDir, { recursive: true }); + const d = discoverWorkspaceRoot({ repos: ["browserstack/testrepo"], from: pluginDir }); + assert.equal(d.root, ws); + assert.equal(d.matched, "browserstack/testrepo"); + assert.equal(d.tried.length, 2, "found on the second candidate"); +}); + +// The bound is a feature: from deep inside a repo the root is out of reach, +// and the correct answer is to stop rather than climb toward `/` and risk +// matching an unrelated checkout. +test("discoverWorkspaceRoot stops at maxTries instead of climbing far", () => { + const deep = join(repoDir, "a", "b", "c"); + mkdirSync(deep, { recursive: true }); + const d = discoverWorkspaceRoot({ repos: ["browserstack/testrepo"], from: deep, maxTries: 3 }); + assert.equal(d.root, null, "workspace is 4 levels up — out of the bounded range"); + assert.equal(d.tried.length, 3); +}); + +// Genericity: a candidate wins only if it holds a repo THIS run validated. +// Nothing about the product or layout is assumed. +test("discoverWorkspaceRoot verifies against the run's own repo list", () => { + const d = discoverWorkspaceRoot({ repos: ["browserstack/some-other-product"], from: repoDir }); + assert.equal(d.root, null, "must not accept a dir that lacks the requested repo"); + assert.match(d.reason, /some-other-product/); +}); + +test("discoverWorkspaceRoot is bounded — it gives up rather than hunting", () => { + const d = discoverWorkspaceRoot({ repos: ["browserstack/nope"], from: repoDir, maxTries: 3 }); + assert.equal(d.root, null); + assert.ok(d.tried.length <= 3, `tried ${d.tried.length}, expected <= 3`); +}); + +test("an explicit root is still VERIFIED, so a stale override fails loudly", () => { + const ok = discoverWorkspaceRoot({ repos: ["browserstack/testrepo"], explicit: ws }); + assert.equal(ok.root, ws); + const bad = discoverWorkspaceRoot({ repos: ["browserstack/testrepo"], explicit: join(ws, "nowhere") }); + assert.equal(bad.root, null, "a wrong explicit path must not be trusted blindly"); +}); + +// This is the context-saving payload: resolved once, read by every coordinator. +test("resolveLocalRepos reports per-repo usability at the pinned sha", () => { + const r = resolveLocalRepos({ + repos: ["browserstack/testrepo", "browserstack/absent"], + pins: { "browserstack/testrepo": sha1 }, + workspaceRoot: ws, + }); + assert.equal(r["browserstack/testrepo"].usable, true); + assert.equal(r["browserstack/testrepo"].sha, sha1); + assert.equal(r["browserstack/absent"].usable, false); + assert.match(r["browserstack/absent"].reason, /no local clone/); +}); + +test("resolveLocalRepos marks a repo unusable when its sha is absent", () => { + const r = resolveLocalRepos({ + repos: ["browserstack/testrepo"], + pins: { "browserstack/testrepo": "0".repeat(40) }, + workspaceRoot: ws, + }); + assert.equal(r["browserstack/testrepo"].usable, false); + assert.match(r["browserstack/testrepo"].reason, /not present locally/); +}); + test("path missing at that commit is a local answer, not a remote fallback", () => { const r = readFileAt({ repo: "browserstack/testrepo", sha: sha1, path: "nope.js", workspaceRoot: ws }); assert.equal(r.ok, false); From d40ed4d4dbb69007b3f322183e3550194d4f8b9c Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:29:18 +0530 Subject: [PATCH 26/51] fix(rca): wire local-repo reading into the gate and the coordinator prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reader worked but nothing invoked it. bin/repo-read.mjs was referenced in zero skill/agent files, and discoverWorkspaceRoot/resolveLocalRepos/ setLocalRepos were called from nowhere outside tests — the whole local-clone path was unreachable at runtime, so every coordinator still went to the network for file contents (31% of gh traffic). Three changes close that: - SKILL.md Step 4 gains a step 6 that resolves the workspace once against the run's validated repo list and persists the per-repo map via setLocalRepos, pinned to deployState commit shas rather than branch names. - ai-tfa-coordinator.md documents repo-read alongside cached-exec/cached-mcp, with the sha-not-branch rule stated as the reason rather than a rule. - repo-read derives the evidence path from the buildId it already receives, so there is no RCA_EVIDENCE_FILE for a dispatch prompt to forget; an explicit override still wins. Verified with no env set: read teststack@41cc211b from the local clone with no network, sourced "via evidence-file (resolved at gate)"; a branch name is still refused. 144 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 10 ++++++++++ bin/repo-read.mjs | 9 ++++++--- skills/rca-build/SKILL.md | 30 +++++++++++++++++++++++++++++- 3 files changed, 45 insertions(+), 4 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 6c3deec..1ad398e 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -124,6 +124,16 @@ read-only and has no side effects, so a read is always safe to repeat. filters share one cached fetch: `node .../cached-exec.mjs "$B" 3895 'gh api repos/o/r/contents/f' | jq -r .content | head -40` One fetch per call — the wrapper refuses `;`/`&&`/backticks/redirects. + - **Repo file contents** — use the repo reader instead of `gh api + .../contents/...` directly. It serves the file from a local clone at the + pinned commit when the gate found one (~37ms, no network), and otherwise + falls through to the same cached `gh` call, so it is never worse: + `node <pluginRoot>/bin/repo-read.mjs <buildId> <testRunId> <org/repo> <sha> <path>` + The `<sha>` MUST be the commit sha from the evidence file's `deployState` + — a **branch name is refused**, because local clones are routinely stale + and would hand you code that never shipped while looking perfectly fine. + Check `localRepos` in the evidence file to see which repos are local; you + do not need to probe the filesystem, the gate already resolved it. - **MCP data queries** (grafana/VictoriaLogs, `listTestIds`, `getFailureLogs`) — check first, and store your digest on a miss: `node <pluginRoot>/bin/cached-mcp.mjs <buildId> get <tool> '<argsJson>'` diff --git a/bin/repo-read.mjs b/bin/repo-read.mjs index dc79169..9f904b6 100644 --- a/bin/repo-read.mjs +++ b/bin/repo-read.mjs @@ -49,10 +49,13 @@ if (!buildId || !writerId || !repo || !sha || !path) { let workspaceRoot = process.env.RCA_WORKSPACE_ROOT; let rootSource = workspaceRoot ? "RCA_WORKSPACE_ROOT" : null; -const evidencePath = process.env.RCA_EVIDENCE_FILE; -if (!workspaceRoot && evidencePath) { +// Derived from the buildId we already have, so there is no env var for a +// dispatch prompt to forget; an explicit override still wins. +if (!workspaceRoot) { try { - const { readEvidenceFile } = await import("../lib/evidence-file.mjs"); + const { readEvidenceFile, evidencePathFor } = await import("../lib/evidence-file.mjs"); + const evidencePath = process.env.RCA_EVIDENCE_FILE + || evidencePathFor(buildId, process.env.RCA_STATE_DIR ?? ""); const lr = readEvidenceFile(evidencePath)?.localRepos; if (lr?.workspaceRoot) { workspaceRoot = lr.workspaceRoot; rootSource = "evidence-file (resolved at gate)"; } } catch { /* evidence file optional */ } diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 3216e88..54fa06c 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -310,7 +310,35 @@ gathers it*. baseline (never-green suite) → fall back to a configured baseline ref and note the weaker grounding — this note travels into the file, not just a spoken log line, so every coordinator sees it. -6. `recomputeCoverage(path, {repos, workloads}, nowMs)` and declare the +6. **Resolve local clones ONCE** (`lib/repo-source.mjs`). File *contents* are + the largest remaining slice of github traffic (31%), and most of it can be + served with no network at all when the machine already has the repos + checked out — measured `git show` ~37ms vs `gh api` ~1022ms for the same + file, byte-identical. + + ```js + const d = discoverWorkspaceRoot({ repos: reposValidated, from: pluginRoot }); + const localRepos = d.root + ? resolveLocalRepos({ repos: reposValidated, pins: deployStateShas, workspaceRoot: d.root }) + : {}; + setLocalRepos(path, { workspaceRoot: d.root, repos: localRepos }, nowMs); + ``` + + `discoverWorkspaceRoot` takes the **validated repo list** and accepts a + candidate directory only if it actually contains one of *this run's* repos — + that check is what keeps the plugin generic, and it is bounded to ~3 tries + because guessing harder risks reading an unrelated checkout, which is + silently wrong rather than merely slow. Finding nothing is a fine outcome: + every read falls back to the cached `gh` path. + + `pins` must be the **build-time commit shas** from `deployState`, never + branch names. A developer's clone is routinely stale (12 commits, measured), + and reading a branch locally returned different bytes than the real head — + for RCA that is a confident wrong answer about code that never shipped. + + Doing this at the gate is the point: every coordinator then reads a map + instead of probing the filesystem itself. +7. `recomputeCoverage(path, {repos, workloads}, nowMs)` and declare the resulting path in the gate summary alongside the capability manifest, so a human re-reading the run can find it. From 867b6a3e5087d6e1e646221cb99cabdd617180fe Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:32:54 +0530 Subject: [PATCH 27/51] fix(rca): reachability guard, folded-view reads, and a staleness signal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes, one of which prevents the other two from recurring. REACHABILITY TEST (tests/wiring.test.mjs). Twice a helper shipped fully unit-tested and invoked by nothing. Unit tests cannot catch it — the module works perfectly alone, which is why it survives review. The prompt layer is the real call graph, so the test asserts every bin/*.mjs is named by some skill/agent markdown, plus that gate-critical lib exports have a call site outside tests. Verified it actually fires: adding an orphan file fails the suite with the helper named. FOLDED READS. Operating Principle 0 told coordinators to `Read` the evidence path — which shows the orchestrator's base file only and hides every contribution shard, the exact bug bin/evidence-show.mjs exists to fix and which cost a real run a re-gather of an 11-PR entry. It now directs --summary/--prs/--repo, and explains why reading the raw JSON is both wrong and more expensive. STALENESS. The file is keyed by buildId alone, so a resumed run silently reuses deployState and a PR window that have both moved on — the same silent-wrong-answer risk we refuse branch names over, with no guard at all. stalenessOf() reports age and advice; it deliberately does not expire anything, since stale build-level context still beats none and the failure window itself never moves. evidence-show warns on stderr on every view, so piped JSON stays clean. Caught while testing on the live build: a FUTURE timestamp clamped to age 0 and reported "fresh" — failing in the reassuring direction on clock skew. Now returns known:false, stale:true. 148 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 25 +++++++++++- bin/evidence-show.mjs | 11 +++++- lib/evidence-file.mjs | 53 +++++++++++++++++++++++++ tests/evidence-file.test.mjs | 41 +++++++++++++++++++ tests/wiring.test.mjs | 76 ++++++++++++++++++++++++++++++++++++ 5 files changed, 203 insertions(+), 3 deletions(-) create mode 100644 tests/wiring.test.mjs diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 1ad398e..be2e006 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -75,8 +75,29 @@ read-only and has no side effects, so a read is always safe to repeat. ## Operating principles -0. **Read the pre-fetch first.** If `evidenceFile` is present, `Read` it - before considering any live github/infra/logs call. It holds build-level +0. **Read the pre-fetch first — through `evidence-show`, not `Read`/`cat`.** + + ```bash + node <pluginRoot>/bin/evidence-show.mjs <evidenceFile> --summary # start here + node <pluginRoot>/bin/evidence-show.mjs <evidenceFile> --prs # falsify by mergedAt + node <pluginRoot>/bin/evidence-show.mjs <evidenceFile> --repo <org/repo> + ``` + + `Read`ing the path directly shows the orchestrator's **base file only** and + silently hides every contribution a prior coordinator wrote, because those + live in per-writer shards. This is not hypothetical: an agent reported "the + file has 2 repos" when the folded view had 5, including an 11-PR entry + another coordinator had already gathered — so it re-did that work. Only + `evidence-show` folds base + shards into the real view. + + Start with `--summary` (one line per repo/workload) and open `--repo` for + the one you need; reading the whole JSON costs tokens for evidence about + failures that aren't yours. `--prs` prints `mergedAt | #num | title`, which + does the most falsification work per byte — anything merged *after* the + build started is disqualified without fetching a single diff (on one real + run that removed 11 of 22 candidates before any `gh pr view`). + + Consult it before considering any live github/infra/logs call. It holds build-level evidence (PR window, deploy state, log sweeps) already gathered once by the orchestrator for the repos/workloads this build's failures implicate. Use what it covers directly — its entries are already digest-shaped (an diff --git a/bin/evidence-show.mjs b/bin/evidence-show.mjs index 7aa5ab8..04f8bed 100644 --- a/bin/evidence-show.mjs +++ b/bin/evidence-show.mjs @@ -15,7 +15,7 @@ // node bin/evidence-show.mjs <evidenceFilePath> --summary # one line per repo/workload // node bin/evidence-show.mjs <evidenceFilePath> --repo <name> -import { readEvidenceFile, readBaseFile, contribDirFor, hasTrustworthyPrList } from "../lib/evidence-file.mjs"; +import { readEvidenceFile, readBaseFile, contribDirFor, hasTrustworthyPrList, stalenessOf } from "../lib/evidence-file.mjs"; import { existsSync, readdirSync } from "node:fs"; const [, , filePath, mode, arg] = process.argv; @@ -26,6 +26,15 @@ if (!filePath) { const folded = readEvidenceFile(filePath); +// Warn on EVERY view, not just --summary. A resumed run reuses this file by +// buildId alone, and deployState/PR-window data keeps moving after it was +// written — the same silent-wrong-answer risk we refuse branch names over. +// stderr, so it never pollutes JSON piped into jq. +{ + const s = stalenessOf(filePath, Date.now()); + if (s.stale || !s.known) console.error(`[evidence-show STALE] ${s.note}`); +} + if (mode === "--repo") { console.log(JSON.stringify(folded.github?.[arg] ?? null, null, 2)); process.exit(0); diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index 604916f..c4b0edd 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -417,3 +417,56 @@ export function recomputeCoverage(filePath, requested, nowMs) { writeEvidenceFile(filePath, doc); return doc.coverage; } + +/** + * How stale is this pre-fetch, and is it still safe to trust? + * + * We are careful never to read a repo at a branch name because a stale clone + * yields a confident wrong answer — but the evidence file had exactly the same + * exposure and no guard at all. It is keyed only by `buildId`, so a + * `pending-resume` row hours or days later silently reuses the original + * `deployState` and PR window. Those describe "what was deployed and what + * merged around the build", and both keep moving after the pre-fetch is + * written. Reusing them blind is the same failure mode, just slower to notice. + * + * This does not expire anything — the file stays usable, because stale + * build-level context is still far better than none and the failure window + * itself never moves. It returns a signal the caller can surface, so a + * coordinator re-verifies a suspect PR instead of trusting a day-old list. + * + * `maxFreshMs` defaults to 6h: comfortably longer than any normal batch run + * (minutes), short enough that an overnight resume is flagged. + */ +export function stalenessOf(filePath, nowMs, maxFreshMs = 6 * 60 * 60 * 1000) { + const doc = readEvidenceFile(filePath); + const generatedAtMs = doc?.generatedAtMs ?? 0; + if (!generatedAtMs) { + return { known: false, stale: false, ageMs: null, note: "no generatedAtMs recorded — age unknown" }; + } + // A timestamp in the FUTURE must never read as "fresh". Clamping the age to + // zero is the tempting one-liner and it fails in the worst direction: clock + // skew between the gate host and a coordinator, or a hand-seeded timestamp, + // would silently certify arbitrarily old evidence as current. We can't tell + // the age, so say so rather than guess in the reassuring direction. + if (generatedAtMs > nowMs) { + return { + known: false, + stale: true, + ageMs: null, + generatedAtMs, + note: `generatedAtMs is ${Math.round((generatedAtMs - nowMs) / 60000)}m in the future (clock skew or a seeded timestamp) — age cannot be trusted; re-verify any PR you are about to name as the cause.`, + }; + } + const ageMs = nowMs - generatedAtMs; + const stale = ageMs > maxFreshMs; + const mins = Math.round(ageMs / 60000); + return { + known: true, + stale, + ageMs, + generatedAtMs, + note: stale + ? `pre-fetch is ${mins}m old (> ${Math.round(maxFreshMs / 60000)}m): deployState and the PR window may have moved since. Still usable — the failure window is fixed — but re-verify any PR you are about to name as the cause.` + : `pre-fetch is ${mins}m old — fresh`, + }; +} diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index e00cc2c..95d3175 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -296,3 +296,44 @@ test("writeEvidenceFile creates the parent directory if missing", () => { writeEvidenceFile(nested, emptyEvidenceFile("build-1", 0)); assert.deepEqual(readEvidenceFile(nested).buildId, "build-1"); }); + +// Staleness: the resume-path analogue of refusing a branch name. +test("stalenessOf flags an old pre-fetch but never invalidates it", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-stale-")); + const { evidencePathFor, initEvidenceFile, stalenessOf, readEvidenceFile } = + await import("../lib/evidence-file.mjs"); + const t0 = 1_700_000_000_000; + const p = evidencePathFor("b-stale", dir); + initEvidenceFile(p, "b-stale", t0); + + const fresh = stalenessOf(p, t0 + 5 * 60 * 1000); + assert.equal(fresh.stale, false, "5m into a run is fresh"); + assert.equal(fresh.known, true); + + const old = stalenessOf(p, t0 + 20 * 60 * 60 * 1000); + assert.equal(old.stale, true, "an overnight resume must be flagged"); + assert.match(old.note, /re-verify/, "must say what to do, not just that it is old"); + + // Crucially it is a SIGNAL, not an expiry — the data is still there, because + // stale build-level context still beats none and the failure window is fixed. + assert.ok(readEvidenceFile(p), "file must remain readable when stale"); + + rmSync(dir, { recursive: true, force: true }); +}); + +// The clamp-to-zero trap: a future timestamp must not read as "fresh". +test("stalenessOf refuses to call a future timestamp fresh", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-skew-")); + const { evidencePathFor, initEvidenceFile, stalenessOf } = await import("../lib/evidence-file.mjs"); + const t0 = 1_700_000_000_000; + const p = evidencePathFor("b-skew", dir); + initEvidenceFile(p, "b-skew", t0); + + // Coordinator's clock is behind the gate's, or the stamp was seeded by hand. + const s = stalenessOf(p, t0 - 11 * 60 * 60 * 1000); + assert.equal(s.stale, true, "unknown age must fail closed, not report fresh"); + assert.equal(s.known, false, "we genuinely cannot compute an age here"); + assert.match(s.note, /future/); + + rmSync(dir, { recursive: true, force: true }); +}); diff --git a/tests/wiring.test.mjs b/tests/wiring.test.mjs new file mode 100644 index 0000000..2b7c004 --- /dev/null +++ b/tests/wiring.test.mjs @@ -0,0 +1,76 @@ +// Guard against SHIPPED-BUT-UNREACHABLE code. +// +// Twice now a helper was built, unit-tested, and manually verified — and then +// invoked by nothing. `bin/repo-read.mjs` and `bin/evidence-show.mjs` were both +// referenced in zero skill/agent files, so at runtime every coordinator kept +// doing the expensive thing the helper existed to avoid. Unit tests can't catch +// this: the module works perfectly in isolation, which is exactly why the gap +// survives review. +// +// The prompt layer IS the call graph here. An agent only runs what its skill or +// agent markdown names, so "is this string mentioned in a prompt file" is the +// real reachability test, crude as it looks. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; + +const ROOT = new URL("..", import.meta.url).pathname; + +/** Every .md under the dirs an agent actually reads. */ +function promptText() { + const out = []; + const walk = (dir) => { + let entries; + try { entries = readdirSync(dir); } catch { return; } + for (const e of entries) { + const p = join(dir, e); + if (statSync(p).isDirectory()) walk(p); + else if (e.endsWith(".md")) out.push(readFileSync(p, "utf8")); + } + }; + for (const d of ["skills", "agents", "workflows", ".claude"]) walk(join(ROOT, d)); + return out.join("\n"); +} + +test("every bin/ helper is named by at least one prompt file", () => { + const prompts = promptText(); + const helpers = readdirSync(join(ROOT, "bin")).filter((f) => f.endsWith(".mjs")); + assert.ok(helpers.length > 0, "expected some helpers to check"); + + const orphans = helpers.filter((h) => !prompts.includes(h)); + assert.deepEqual( + orphans, + [], + `unreachable helper(s): ${orphans.join(", ")}. A helper no skill or agent ` + + `names will never run — either reference it from the prompt layer or delete it.`, + ); +}); + +// The exported-but-uncalled variant of the same bug: lib functions that exist +// only because a test calls them. Checked for the few whose whole purpose is to +// be driven by the gate, where being uncalled means the feature is off. +test("gate-critical lib exports are actually invoked outside tests", () => { + const prompts = promptText(); + const src = []; + const walk = (dir) => { + for (const e of readdirSync(dir)) { + const p = join(dir, e); + if (statSync(p).isDirectory()) walk(p); + else if (e.endsWith(".mjs")) src.push(readFileSync(p, "utf8")); + } + }; + walk(join(ROOT, "lib")); + walk(join(ROOT, "bin")); + const haystack = src.join("\n") + "\n" + prompts; + + // Each of these is a no-op unless something drives it: discovery that is + // never run means every read falls back to the network, and a map that is + // never written means every coordinator re-probes the filesystem. + for (const fn of ["discoverWorkspaceRoot", "resolveLocalRepos", "setLocalRepos", "recomputeCoverage"]) { + // Definition line doesn't count as a call site. + const uses = haystack.split(fn).length - 1; + assert.ok(uses >= 2, `${fn} appears ${uses}x outside tests — defined but never driven`); + } +}); From 220c7ba5eef3d4f57e7661764c7431501917a21b Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:37:57 +0530 Subject: [PATCH 28/51] fix(rca): refuse a foreign state-CSV schema instead of silently dropping columns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit readRows keyed rows by whatever header the file had, while writeRows only ever emits COLUMNS — so any header name not in COLUMNS was silently discarded on the next write. Found on a real legacy 10-column state file (test_id,test_name,error_summary,is_flaky,...): flipping one row through the current code returned "resolved" and reported success, having dropped test_id and test_name entirely and blanked cluster_id. Losing which test a row describes is worse than any error we could raise. readRows now normalises the header through COLUMN_ALIASES (previously applied only to flip()'s field names, not to the header) and throws on anything still unrecognised, naming the offending columns and saying to re-seed. Known legacy spellings — test_run_id, status, thread_id, turn_id — keep working. This is why build fiehz... cannot simply be resumed: its CSV predates the current schema and must be re-seeded. 150 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/csv-state.mjs | 20 +++++++++++++++++++- tests/csv-state.test.mjs | 32 +++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index ee970c5..fe6a629 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -135,7 +135,25 @@ export function readRows(csvPath) { const text = readFileSync(csvPath, "utf8"); const raw = parseCsv(text).filter((r) => r.some((c) => c.length > 0)); if (raw.length === 0) return []; - const header = raw[0]; + // Normalise the HEADER, not just flip()'s field names. writeRows only ever + // emits COLUMNS, so any header name we fail to recognise here is silently + // dropped the next time the file is written. That is not theoretical: a + // legacy 10-column state file (`test_id,test_name,…`) round-tripped through + // flip() came back with test_id and test_name gone and cluster_id blanked, + // reporting success the whole way. Losing which test a row describes is + // worse than any error we could raise. + const header = raw[0].map((c) => COLUMN_ALIASES.get(c) ?? c); + const unknown = header.filter((c) => c && !COLUMNS.includes(c)); + if (unknown.length) { + // Loud, and it names the columns — a foreign schema means this file was + // written by a different version, and guessing an alignment for it would + // reintroduce exactly the silent corruption above. + throw new Error( + `[csv-state] ${csvPath} has ${unknown.length} unrecognised column(s): ${unknown.join(", ")}. ` + + `This file was written by a different schema version; writing it back would DROP those columns. ` + + `Re-seed the build instead of resuming this file.`, + ); + } return raw.slice(1).map((cells) => { const row = {}; header.forEach((col, idx) => { diff --git a/tests/csv-state.test.mjs b/tests/csv-state.test.mjs index b5caeb3..24b042c 100644 --- a/tests/csv-state.test.mjs +++ b/tests/csv-state.test.mjs @@ -1,6 +1,6 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -207,3 +207,33 @@ test("csvPathFor: stateDir override wins over temp", () => { const p = csvPathFor("b1", "/ci/artifacts"); assert.equal(p, join("/ci/artifacts", "rca-state.b1.csv")); }); + +// A foreign header must fail loudly, because writeRows only emits COLUMNS and +// would silently drop anything it didn't recognise. A real legacy 10-column +// file lost test_id and test_name this way while reporting success. +test("readRows refuses a foreign schema instead of silently dropping columns", () => { + const dir = mkdtempSync(join(tmpdir(), "rca-legacy-")); + const p = join(dir, "legacy.csv"); + writeFileSync(p, "test_id,test_name,rca_done\nt1,login spec,pending\n", "utf8"); + + assert.throws(() => readRows(p), /unrecognised column/i, + "must name the problem rather than mangle the file"); + assert.throws(() => readRows(p), /test_id/, "must say WHICH columns"); + + rmSync(dir, { recursive: true, force: true }); +}); + +// Known legacy spellings are still accepted — the guard is for genuinely +// foreign schemas, not for every older name. +test("readRows maps aliased header names rather than rejecting them", () => { + const dir = mkdtempSync(join(tmpdir(), "rca-alias-")); + const p = join(dir, "aliased.csv"); + writeFileSync(p, "test_run_id,status,thread_id\n42,pending,th-1\n", "utf8"); + + const rows = readRows(p); + assert.equal(rows[0].testRunId, "42"); + assert.equal(rows[0].rca_done, "pending"); + assert.equal(rows[0].threadId, "th-1"); + + rmSync(dir, { recursive: true, force: true }); +}); From 3f015a699fa94eb46961f76e50ceba576504e3e1 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 14:58:44 +0530 Subject: [PATCH 29/51] fix(rca): tighten a pre-existing state dir, not just newly created ones MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mkdirSync's `mode` applies only when it creates the directory — the same trap already fixed for the files themselves. Found on this machine: <tmpdir>/bstack-rca was drwxr-xr-x, created before the hardening landed, with correctly-0600 files inside it. The directory listing alone leaks which builds were analysed, and pre-hardening files inside it are still 0644. These artifacts hold root causes, culprit PRs and log excerpts and live in a shared OS temp dir, so ensureOwnerOnlyDir() now chmods an existing directory to 0700 as well as creating new ones that way. Shared by csv-state, evidence-file and tool-cache, which all had the identical pattern. 151 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/csv-state.mjs | 16 +++++++++++++++- lib/evidence-file.mjs | 22 ++++++++++++++++++++-- lib/tool-cache.mjs | 12 +++++++++++- tests/csv-state.test.mjs | 20 +++++++++++++++++++- 4 files changed, 65 insertions(+), 5 deletions(-) diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index fe6a629..69c27e8 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -167,13 +167,27 @@ export function readRows(csvPath) { // OS temp dir and records root causes, culprit PRs and evidence digests. export function writeRows(csvPath, rows) { const dir = dirname(csvPath); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + // `mode` applies on CREATE only — the same trap already fixed for the files + // themselves. A directory made before hardening stays 0755 forever, which on + // a shared machine leaves root causes, culprit PRs and log excerpts readable + // by every local user. Tighten an existing one too. + if (dir) ensureOwnerOnlyDir(dir); const existed = existsSync(csvPath); writeFileSync(csvPath, encodeRows(rows), { encoding: "utf8", mode: 0o600 }); // `mode` applies on create only — tighten a pre-hardening leftover too. if (existed) chmodSync(csvPath, 0o600); } +// Owner-only, on create AND on an existing directory. `mkdirSync`'s `mode` +// applies only when it creates the dir, so one made before this hardening +// landed keeps its 0755 forever — and these artifacts hold root causes, +// culprit PRs and log excerpts in a shared OS temp dir. Found in practice: +// <tmpdir>/bstack-rca was drwxr-xr-x with 0600 files inside it. +function ensureOwnerOnlyDir(dir) { + if (!existsSync(dir)) { mkdirSync(dir, { recursive: true, mode: 0o700 }); return; } + try { chmodSync(dir, 0o700); } catch { /* not ours to tighten; leave it */ } +} + function emptyRow() { return Object.fromEntries(COLUMNS.map((c) => [c, ""])); } diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index c4b0edd..d7fff41 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -104,6 +104,16 @@ function readContribs(basePath) { return out; } +// Owner-only, on create AND on an existing directory. `mkdirSync`'s `mode` +// applies only when it creates the dir, so one made before this hardening +// landed keeps its 0755 forever — and these artifacts hold root causes, +// culprit PRs and log excerpts in a shared OS temp dir. Found in practice: +// <tmpdir>/bstack-rca was drwxr-xr-x with 0600 files inside it. +function ensureOwnerOnlyDir(dir) { + if (!existsSync(dir)) { mkdirSync(dir, { recursive: true, mode: 0o700 }); return; } + try { chmodSync(dir, 0o700); } catch { /* not ours to tighten; leave it */ } +} + export function emptyEvidenceFile(buildId, nowMs) { return { buildId: String(buildId ?? ""), @@ -203,7 +213,11 @@ export function readEvidenceFile(filePath) { // and carries private-repo PR detail and app-log digests. export function writeEvidenceFile(filePath, doc) { const dir = dirname(filePath); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + // `mode` applies on CREATE only — the same trap already fixed for the files + // themselves. A directory made before hardening stays 0755 forever, which on + // a shared machine leaves root causes, culprit PRs and log excerpts readable + // by every local user. Tighten an existing one too. + if (dir) ensureOwnerOnlyDir(dir); const existed = existsSync(filePath); writeFileSync(filePath, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); // `mode` is only honoured when the file is CREATED. A file left over from a @@ -308,7 +322,11 @@ function loadOwnShard(basePath, writerId, nowMs) { function writeShard(path, doc) { const dir = dirname(path); - if (dir && !existsSync(dir)) mkdirSync(dir, { recursive: true, mode: 0o700 }); + // `mode` applies on CREATE only — the same trap already fixed for the files + // themselves. A directory made before hardening stays 0755 forever, which on + // a shared machine leaves root causes, culprit PRs and log excerpts readable + // by every local user. Tighten an existing one too. + if (dir) ensureOwnerOnlyDir(dir); writeFileSync(path, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); } diff --git a/lib/tool-cache.mjs b/lib/tool-cache.mjs index e77f71f..268e78f 100644 --- a/lib/tool-cache.mjs +++ b/lib/tool-cache.mjs @@ -49,6 +49,16 @@ export function toolCacheDirFor(buildId, stateDir = "") { * formatting of the same command still hits, but nothing else is rewritten — * `| head -20` vs `| head -200` genuinely return different output and must * stay distinct keys. */ +// Owner-only, on create AND on an existing directory. `mkdirSync`'s `mode` +// applies only when it creates the dir, so one made before this hardening +// landed keeps its 0755 forever — and these artifacts hold root causes, +// culprit PRs and log excerpts in a shared OS temp dir. Found in practice: +// <tmpdir>/bstack-rca was drwxr-xr-x with 0600 files inside it. +function ensureOwnerOnlyDir(dir) { + if (!existsSync(dir)) { mkdirSync(dir, { recursive: true, mode: 0o700 }); return; } + try { chmodSync(dir, 0o700); } catch { /* not ours to tighten; leave it */ } +} + export function cacheKey(command) { const norm = String(command ?? "").replace(/\s+/g, " ").trim(); return createHash("sha256").update(norm).digest("hex").slice(0, 24); @@ -350,7 +360,7 @@ export function cachePut(cacheDir, key, entry, nowMs) { // its own permissions, since silently chmod'ing a path we were handed would // be presumptuous. Entries stay 0600 regardless, and a traversable directory // only exposes opaque hash filenames, not their contents. - if (!existsSync(cacheDir)) mkdirSync(cacheDir, { recursive: true, mode: 0o700 }); + ensureOwnerOnlyDir(cacheDir); const raw = redact(entry.stdout ?? ""); const truncated = raw.length > MAX_BYTES; const rec = { diff --git a/tests/csv-state.test.mjs b/tests/csv-state.test.mjs index 24b042c..0dad786 100644 --- a/tests/csv-state.test.mjs +++ b/tests/csv-state.test.mjs @@ -1,12 +1,13 @@ import { test, beforeEach, afterEach } from "node:test"; import assert from "node:assert/strict"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync, mkdirSync, chmodSync, statSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { csvPathFor, seed, readRows, + writeRows, claim, heartbeat, flip, @@ -237,3 +238,20 @@ test("readRows maps aliased header names rather than rejecting them", () => { rmSync(dir, { recursive: true, force: true }); }); + +// mkdirSync's `mode` applies on CREATE only, so a directory made before the +// hardening landed keeps 0755 forever — with root causes and culprit PRs in it. +test("writeRows tightens a pre-existing world-readable state dir", () => { + const dir = mkdtempSync(join(tmpdir(), "rca-perm-")); + const loose = join(dir, "loose"); + mkdirSync(loose, { mode: 0o755 }); + chmodSync(loose, 0o755); // as an older version would have left it + + const csv = join(loose, "rca-state.b.csv"); + writeRows(csv, []); + + assert.equal(statSync(loose).mode & 0o777, 0o700, "existing dir must be tightened, not left open"); + assert.equal(statSync(csv).mode & 0o777, 0o600); + + rmSync(dir, { recursive: true, force: true }); +}); From 95539f1a99d33ebdb6ca8a03d47d3d1649b4a730 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 15:11:39 +0530 Subject: [PATCH 30/51] feat(rca): sweep the state dir at gate startup; add an explicit prune MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-write hardening only tightens the file being written, so artifacts from a build analysed before that landed keep their old permissions forever — a completed build is never rewritten again. On this machine that left 6 files at 0644 holding root causes, culprit PRs and log excerpts in a shared OS temp dir. Fixing it needs a sweep, not another per-write guard. hardenStateDir() walks the tree and makes it owner-only (0700 dirs / 0600 files). Idempotent and cheap, so Step 2 now runs it unconditionally. It never throws: a file owned by another user is skipped and counted, because failing an RCA run over one un-chmod-able leftover is worse than the leak it closes. pruneStateDir() addresses growth (2.2MB / 143 files / 6 builds accumulated here, nothing ever deleting them). It is deliberately NOT automatic and defaults to 7 days: these files ARE the resume state, keyed by buildId, so anything that deletes them can silently turn a resumable build into a lost one. dryRun reports without touching. Uses mtime, not atime, so reading a file during a resume doesn't make an abandoned build look fresh. Applied to the live directory: 5 dirs / 143 files swept, 6 leftovers repaired, 0 skipped; prune dry-run confirms nothing is old enough to remove. Two test bugs found and fixed while writing this: a hardcoded future nowMs made the whole fixture look ancient (passing for the wrong reason), and `new Date(seconds)` was read as milliseconds, landing every stamp in 1970. 155 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/state-dir.mjs | 136 ++++++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 14 ++++ tests/state-dir.test.mjs | 102 ++++++++++++++++++++++++++++ 3 files changed, 252 insertions(+) create mode 100644 lib/state-dir.mjs create mode 100644 tests/state-dir.test.mjs diff --git a/lib/state-dir.mjs b/lib/state-dir.mjs new file mode 100644 index 0000000..d91570e --- /dev/null +++ b/lib/state-dir.mjs @@ -0,0 +1,136 @@ +// Housekeeping for the shared state directory (`<tmpdir>/bstack-rca/`). +// +// Everything a run produces — the state CSV, the evidence file and its +// contribution shards, the tool cache — lands here and is NEVER deleted by the +// run itself. That is deliberate: resume is keyed on buildId → same path, so +// cleaning up on completion would break `pending-resume`. The cost is that the +// directory accumulates, and that artifacts written by older versions keep +// whatever permissions they were created with. +// +// Both problems need a sweep rather than a per-write fix, because a write only +// ever touches the one file it is writing. `hardenStateDir` is cheap enough to +// run unconditionally at gate startup; `pruneStateDir` is deliberately NOT +// automatic (see below). + +import { existsSync, readdirSync, statSync, chmodSync, rmSync } from "node:fs"; +import { join } from "node:path"; + +/** + * Make the whole state tree owner-only, repairing anything left open by an + * older version. + * + * Per-write hardening can't do this: `writeRows` tightens the file it writes + * and nothing else, so a build analysed before the hardening landed keeps its + * 0644 forever unless something rewrites it — and a completed build never gets + * rewritten. Measured on a real machine: the directory itself was drwxr-xr-x + * and 6 files were still 0644, holding root causes, culprit PRs and log + * excerpts in a shared OS temp dir. + * + * Never throws: a file owned by another user is skipped, because failing the + * whole RCA run over one un-chmod-able leftover would be a worse outcome than + * the leak we're closing. + * + * Returns `{ dirs, files, skipped }` counts. + */ +export function hardenStateDir(dir) { + const out = { dirs: 0, files: 0, skipped: [] }; + if (!dir || !existsSync(dir)) return out; + + const walk = (p) => { + let st; + try { + st = statSync(p); + } catch { + out.skipped.push(p); + return; + } + const isDir = st.isDirectory(); + const want = isDir ? 0o700 : 0o600; + if ((st.mode & 0o777) !== want) { + try { + chmodSync(p, want); + } catch { + out.skipped.push(p); + return; // can't chmod it; don't pretend we descended into it either + } + } + if (isDir) { + out.dirs++; + let entries = []; + try { + entries = readdirSync(p); + } catch { + out.skipped.push(p); + return; + } + for (const e of entries) walk(join(p, e)); + } else { + out.files++; + } + }; + + walk(dir); + return out; +} + +/** + * Delete build artifacts older than `maxAgeMs` (default 7 days). + * + * NOT called automatically, and the default is deliberately far longer than + * any run: these files ARE the resume state, so anything that deletes them can + * silently turn a resumable build into a lost one. Seven days is well past the + * minutes a batch takes while still bounding growth, and the caller has to ask + * for it explicitly. + * + * `dryRun: true` reports what would go without touching anything — use it + * before wiring this into anything automatic. + * + * Returns `{ removed, bytes, kept, dryRun }`. + */ +export function pruneStateDir(dir, nowMs, { maxAgeMs = 7 * 24 * 60 * 60 * 1000, dryRun = false } = {}) { + const res = { removed: [], bytes: 0, kept: 0, dryRun }; + if (!dir || !existsSync(dir)) return res; + + const sizeOf = (p) => { + let total = 0; + const st = statSync(p); + if (!st.isDirectory()) return st.size; + for (const e of readdirSync(p)) { + try { + total += sizeOf(join(p, e)); + } catch { /* vanished mid-walk */ } + } + return total; + }; + + for (const name of readdirSync(dir)) { + const p = join(dir, name); + let st; + try { + st = statSync(p); + } catch { + continue; + } + // mtime, not atime: reading an evidence file during a resume should not + // make a long-abandoned build look freshly relevant. + if (nowMs - st.mtimeMs <= maxAgeMs) { + res.kept++; + continue; + } + let bytes = 0; + try { + bytes = sizeOf(p); + } catch { /* best effort */ } + if (!dryRun) { + try { + rmSync(p, { recursive: true, force: true }); + } catch { + res.kept++; + continue; + } + } + res.removed.push(name); + res.bytes += bytes; + } + return res; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 54fa06c..20ba877 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -206,6 +206,20 @@ listTestIds(buildId=<id>, status="failed", includeFailureDetail=true) (`failure.{category, error_summary, file_path, …}`) — the seed for clustering, so no per-test probe turns are needed. +**First, sweep the state directory** (`lib/state-dir.mjs` → `hardenStateDir(dir)`). +Per-write hardening only tightens the file being written, so artifacts from a +build analysed before that landed keep their old permissions forever — a +completed build is never rewritten. Found in practice: the directory itself was +`drwxr-xr-x` with six `0644` files inside, holding root causes, culprit PRs and +log excerpts in a shared OS temp dir. The sweep is cheap and idempotent, so run +it unconditionally; it never throws, skipping anything it cannot chmod. + +Nothing deletes these artifacts when a run finishes, and that is deliberate — +resume is keyed on `buildId` → same path, so cleaning up on completion would +break `pending-resume`. `pruneStateDir(dir, nowMs)` exists for growth (default +7 days, far longer than any run) but is **not** automatic: these files *are* the +resume state. Call it explicitly, with `dryRun: true` first. + Resolve the state file with `lib/csv-state.mjs` → `csvPathFor(buildId, config.paths.stateDir)` — the **build id is in the filename** and the default directory is **OS temp** (`<tmpdir>/bstack-rca/rca-state.<buildId>.csv`), so diff --git a/tests/state-dir.test.mjs b/tests/state-dir.test.mjs new file mode 100644 index 0000000..1613785 --- /dev/null +++ b/tests/state-dir.test.mjs @@ -0,0 +1,102 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync, mkdirSync, writeFileSync, chmodSync, statSync, existsSync, utimesSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { hardenStateDir, pruneStateDir } from "../lib/state-dir.mjs"; + +const mode = (p) => statSync(p).mode & 0o777; + +function fixture() { + const root = mkdtempSync(join(tmpdir(), "rca-sd-")); + const dir = join(root, "bstack-rca"); + mkdirSync(dir, { mode: 0o755 }); + writeFileSync(join(dir, "rca-state.b1.csv"), "a\n", { mode: 0o644 }); + const cache = join(dir, "rca-toolcache.b1"); + mkdirSync(cache, { mode: 0o755 }); + writeFileSync(join(cache, "entry.json"), "{}", { mode: 0o644 }); + chmodSync(dir, 0o755); + chmodSync(cache, 0o755); + return { root, dir, cache }; +} + +// Per-write hardening only fixes the file being written, so a build analysed +// before the hardening landed keeps 0644 forever — a completed build is never +// rewritten. This is the sweep that repairs them. +test("hardenStateDir tightens leftovers recursively", () => { + const { root, dir, cache } = fixture(); + assert.equal(mode(dir), 0o755, "fixture must start open, else the test proves nothing"); + + const r = hardenStateDir(dir); + + assert.equal(mode(dir), 0o700); + assert.equal(mode(join(dir, "rca-state.b1.csv")), 0o600); + assert.equal(mode(cache), 0o700, "nested cache dir too"); + assert.equal(mode(join(cache, "entry.json")), 0o600, "files inside nested dirs too"); + assert.equal(r.files, 2); + assert.equal(r.dirs, 2); + + rmSync(root, { recursive: true, force: true }); +}); + +test("hardenStateDir is idempotent and safe on a missing dir", () => { + const { root, dir } = fixture(); + hardenStateDir(dir); + const second = hardenStateDir(dir); + assert.equal(mode(dir), 0o700); + assert.equal(second.files, 2, "still walks, just has nothing to change"); + + assert.deepEqual(hardenStateDir(join(root, "nope")), { dirs: 0, files: 0, skipped: [] }); + rmSync(root, { recursive: true, force: true }); +}); + +// These files ARE the resume state, so the default must not be able to eat a +// build someone is about to resume. +test("pruneStateDir keeps recent artifacts and removes only old ones", () => { + const { root, dir } = fixture(); + // Pin every fixture file's mtime relative to the test's own clock — the + // files are created at real "now", so a hardcoded future nowMs would make + // the whole fixture look ancient and the test would pass for the wrong + // reason (it did, first time round). + const now = 1_800_000_000_000; + // utimesSync takes SECONDS as a plain number; `new Date(seconds)` would be + // read as milliseconds and land every stamp in 1970. + const stamp = (p, ageSec) => { const s = now / 1000 - ageSec; utimesSync(p, s, s); }; + stamp(join(dir, "rca-state.b1.csv"), 60); + stamp(join(dir, "rca-toolcache.b1"), 60); + const old = join(dir, "rca-state.ancient.csv"); + writeFileSync(old, "x\n"); + stamp(old, 8 * 24 * 60 * 60); + + const r = pruneStateDir(dir, now); + + assert.deepEqual(r.removed, ["rca-state.ancient.csv"]); + assert.equal(existsSync(old), false); + assert.ok(existsSync(join(dir, "rca-state.b1.csv")), "a fresh build must survive"); + assert.ok(r.kept >= 1); + + rmSync(root, { recursive: true, force: true }); +}); + +test("pruneStateDir dryRun reports without deleting", () => { + const { root, dir } = fixture(); + // Pin every fixture file's mtime relative to the test's own clock — the + // files are created at real "now", so a hardcoded future nowMs would make + // the whole fixture look ancient and the test would pass for the wrong + // reason (it did, first time round). + const now = 1_800_000_000_000; + // utimesSync takes SECONDS as a plain number; `new Date(seconds)` would be + // read as milliseconds and land every stamp in 1970. + const stamp = (p, ageSec) => { const s = now / 1000 - ageSec; utimesSync(p, s, s); }; + stamp(join(dir, "rca-state.b1.csv"), 60); + stamp(join(dir, "rca-toolcache.b1"), 60); + const old = join(dir, "rca-state.ancient.csv"); + writeFileSync(old, "x\n"); + stamp(old, 8 * 24 * 60 * 60); + + const r = pruneStateDir(dir, now, { dryRun: true }); + assert.deepEqual(r.removed, ["rca-state.ancient.csv"]); + assert.equal(existsSync(old), true, "dryRun must not delete"); + + rmSync(root, { recursive: true, force: true }); +}); From 2e4e7140c7892ce0fd9a6a0bd1b0a4f088256228 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 15:18:42 +0530 Subject: [PATCH 31/51] fix(rca): two gate bugs found by an actual dry run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both only surface when the gate runs for real; neither is visible to unit tests. CONNECTOR DISCOVERY MISSED EVERY CONNECTOR. Step 0 ran `ls .claude/skills/ ~/.claude/skills/`. When the plugin is itself a repo inside the workspace, cwd is the plugin and the product's connector skills sit one or two levels UP — so the listing found nothing and the run would have degraded to raw MCP tools with best-effort repo guesses, exactly the failure mode Step 0 exists to prevent. It missed all three real connectors here (tra-regression-context, nl2steps-github, nl2steps-infra). Now also checks ../ and ../../. PINS EXISTED ONLY AS PROSE. resolveLocalRepos needs {repo: sha}, but Step 4 wrote the build-time sha into deployState.summary as English ("Branch tip on <branch> at build start = cd88535b"). There was no structured field, so the gate got an empty pin map and every repo fell back to the network while appearing to work — 0 of 6 local instead of 3 of 6. deployShas() prefers an explicit deployState.sha and falls back to parsing the summary, reporting which happened. The fallback anchors on the build-start phrase: a bare hex word would also match the deploy timestamp 260731135020Z sitting in the same sentence. Verified end to end on build fiehz…: workspace found in 2 tries, 3 of 6 repos resolved local, and a coordinator read a 280KB file with no network. 156 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/evidence-file.mjs | 38 ++++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 17 ++++++++++++++-- tests/evidence-file.test.mjs | 29 +++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 2 deletions(-) diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index d7fff41..d5f39e6 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -488,3 +488,41 @@ export function stalenessOf(filePath, nowMs, maxFreshMs = 6 * 60 * 60 * 1000) { : `pre-fetch is ${mins}m old — fresh`, }; } + +/** + * The build-time commit sha per repo, as a structured map — the input + * `resolveLocalRepos` needs for its `pins`. + * + * Step 4 SHOULD set `deployState.sha` explicitly. It historically didn't: the + * sha lived only in the prose `summary` ("Branch tip on <branch> at build + * start = cd88535b (deploy proxy)"), so the one consumer that needs it + * structurally had to regex English, and got an empty map when the wording + * drifted. That silently downgraded every local read to a network call while + * looking like it worked. + * + * So: prefer the explicit field, fall back to parsing the summary, and report + * which happened so a caller can tell "no sha recorded" from "sha recovered + * from prose". A bare 7-40 hex word is NOT enough on its own — timestamps and + * image tags match that too — so the fallback anchors on an `=`/`:` after a + * build-start phrase. + */ +export function deployShas(filePathOrDoc) { + const doc = typeof filePathOrDoc === "string" ? readEvidenceFile(filePathOrDoc) : filePathOrDoc; + const pins = {}; + const source = {}; + for (const [repo, entry] of Object.entries(doc?.github ?? {})) { + const ds = entry?.deployState; + if (!ds) continue; + if (typeof ds.sha === "string" && /^[0-9a-f]{7,40}$/i.test(ds.sha)) { + pins[repo] = ds.sha; + source[repo] = "field"; + continue; + } + const m = String(ds.summary ?? "").match(/at build start\s*[=:]\s*([0-9a-f]{7,40})\b/i); + if (m) { + pins[repo] = m[1]; + source[repo] = "parsed-from-summary"; + } + } + return { pins, source }; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 20ba877..ced942a 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -48,7 +48,13 @@ pass. The gate has two parts; both run before any RCA work starts. Run: ```bash -ls .claude/skills/ ~/.claude/skills/ 2>/dev/null +# cwd, the WORKSPACE ROOT above it, and the user dir. The middle one matters: +# when this plugin is itself a repo inside the workspace, cwd is the plugin and +# the product's connector skills sit one or two levels UP, so a bare +# `ls .claude/skills/` finds nothing and the run silently degrades to raw MCP +# tools with best-effort repo guesses. Measured: it missed all three real +# connectors on this workspace. +ls .claude/skills/ ../.claude/skills/ ../../.claude/skills/ ~/.claude/skills/ 2>/dev/null ``` For each `SKILL.md` found, open it and look for a **Capability declaration** @@ -332,8 +338,9 @@ gathers it*. ```js const d = discoverWorkspaceRoot({ repos: reposValidated, from: pluginRoot }); + const { pins } = deployShas(path); // structured, not prose const localRepos = d.root - ? resolveLocalRepos({ repos: reposValidated, pins: deployStateShas, workspaceRoot: d.root }) + ? resolveLocalRepos({ repos: reposValidated, pins, workspaceRoot: d.root }) : {}; setLocalRepos(path, { workspaceRoot: d.root, repos: localRepos }, nowMs); ``` @@ -345,6 +352,12 @@ gathers it*. silently wrong rather than merely slow. Finding nothing is a fine outcome: every read falls back to the cached `gh` path. + Set `deployState.sha` explicitly when you write each repo's entry. + `deployShas()` falls back to parsing the prose `summary`, but that is a + safety net, not the contract: when the wording drifts it returns an empty + map, and every read silently degrades to the network while still looking + like it worked. + `pins` must be the **build-time commit shas** from `deployState`, never branch names. A developer's clone is routinely stale (12 commits, measured), and reading a branch locally returned different bytes than the real head — diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index 95d3175..d46f621 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -337,3 +337,32 @@ test("stalenessOf refuses to call a future timestamp fresh", async () => { rmSync(dir, { recursive: true, force: true }); }); + +// The sha lived only in prose, so the one consumer that needs it structurally +// got an empty map — silently downgrading every local read to a network call. +test("deployShas prefers the explicit field and falls back to the summary", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-pins-")); + const { evidencePathFor, initEvidenceFile, setGithubEvidence, deployShas } = + await import("../lib/evidence-file.mjs"); + const p = evidencePathFor("b-pins", dir); + initEvidenceFile(p, "b-pins", 1); + + setGithubEvidence(p, "org/explicit", { deployState: { sha: "abc1234", summary: "" } }, 2); + setGithubEvidence(p, "org/prose", { + deployState: { summary: "Branch tip on main at build start = cd88535b (deploy proxy). Redeploy stamped 260731135020Z." }, + }, 3); + setGithubEvidence(p, "org/none", { deployState: { summary: "no sha here" } }, 4); + + const { pins, source } = deployShas(p); + assert.equal(pins["org/explicit"], "abc1234"); + assert.equal(source["org/explicit"], "field"); + assert.equal(pins["org/prose"], "cd88535b", "must recover the sha from prose"); + assert.equal(source["org/prose"], "parsed-from-summary"); + assert.equal(pins["org/none"], undefined, "absent must stay absent, not guess"); + + // The timestamp 260731135020Z is hex-ish and long — anchoring on the + // build-start phrase is what stops it being mistaken for a commit. + assert.notEqual(pins["org/prose"], "260731135020"); + + rmSync(dir, { recursive: true, force: true }); +}); From 8c60a83c8a52c67332272352a045184e8d28f40d Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 15:25:31 +0530 Subject: [PATCH 32/51] fix(rca): stop numberless PRs collapsing into a single evidence entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both fold paths deduped prsInWindow with `String(p.pr)` as the key. When a contributor writes back PRs without a `pr` number, every one of them keys to the literal string "undefined" and the list silently collapses to whichever arrived last. Caught during a live dry run, not by a test: a coordinator reported writing a 6-PR BStackAutomation window for its 21 siblings; the file kept 1, with `pr: undefined`, and still flagged the entry `prsSearched: true` — so the gap read as closed while five PRs had vanished. Siblings would have inherited a confidently-wrong window. prKey() now falls back to url, then title, then a per-entry anonymous key, so unnumbered PRs stay distinct and two unknowns are never treated as the same PR. Numbered PRs still merge across writers, and '#10' and 10 normalise to the same key. 158 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/evidence-file.mjs | 25 +++++++++++++++++++---- tests/evidence-file.test.mjs | 39 ++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 4 deletions(-) diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index d5f39e6..676cda5 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -153,6 +153,23 @@ function pickLeaf(base, incoming) { return incoming; } +// Dedupe key for a PR entry. Union-by-number is right ONLY when a number is +// present: `String(undefined)` is the constant "undefined", so every +// numberless PR collides on one key and the list silently collapses to the +// last one. Observed live — a coordinator wrote back a 6-PR window and the +// file kept 1, with `pr: undefined`, while still reporting the search as +// trustworthy. Fall back to a content key so unnumbered entries survive +// distinctly, and never treat two unknowns as the same PR. +function prKey(pr, index) { + const n = pr?.pr; + if (n !== undefined && n !== null && String(n).trim() !== "" && String(n) !== "undefined") { + return `#${String(n).replace(/^#/, "")}`; + } + const t = String(pr?.title ?? "").trim(); + const u = String(pr?.url ?? pr?.link ?? "").trim(); + return u ? `url:${u}` : t ? `title:${t}` : `anon:${index}`; +} + function foldGithub(target, repo, entry) { const cur = target[repo] ?? { deployState: null, prsInWindow: [], gap: null }; const next = { @@ -165,8 +182,8 @@ function foldGithub(target, repo, entry) { gap: cur.gap ?? null, }; if (Array.isArray(entry.prsInWindow)) { - const byPr = new Map((next.prsInWindow ?? []).map((p) => [String(p.pr), p])); - for (const pr of entry.prsInWindow) byPr.set(String(pr.pr), pr); + const byPr = new Map((next.prsInWindow ?? []).map((p, i) => [prKey(p, i), p])); + entry.prsInWindow.forEach((pr, i) => byPr.set(prKey(pr, `in-${i}`), pr)); next.prsInWindow = [...byPr.values()]; } // A contributor supplying real content clears the pre-fetch's gap. @@ -341,8 +358,8 @@ export function contributeGithubEvidence(basePath, writerId, repo, patch, nowMs) const entry = doc.github[repo] ?? { deployState: null, prsInWindow: [], gap: null }; if (patch.deployState !== undefined) entry.deployState = patch.deployState; if (Array.isArray(patch.prsInWindow)) { - const byPr = new Map((entry.prsInWindow ?? []).map((p) => [String(p.pr), p])); - for (const pr of patch.prsInWindow) byPr.set(String(pr.pr), pr); + const byPr = new Map((entry.prsInWindow ?? []).map((p, i) => [prKey(p, i), p])); + patch.prsInWindow.forEach((pr, i) => byPr.set(prKey(pr, `in-${i}`), pr)); entry.prsInWindow = [...byPr.values()]; // Contributing a list — even an empty one — means you actually ran the // search, so record that. Pass `prsSearched: false` explicitly to opt out. diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index d46f621..db2bf8f 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -366,3 +366,42 @@ test("deployShas prefers the explicit field and falls back to the summary", asyn rmSync(dir, { recursive: true, force: true }); }); + +// Observed live: a coordinator wrote back a 6-PR window and the file kept ONE, +// with `pr: undefined`, while still flagging the search trustworthy. Cause: +// String(undefined) is the constant "undefined", so every numberless PR +// collided on a single dedupe key. +test("numberless PRs do not collapse into one another", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-prkey-")); + const { evidencePathFor, initEvidenceFile, contributeGithubEvidence, readEvidenceFile } = + await import("../lib/evidence-file.mjs"); + const p = evidencePathFor("b-prkey", dir); + initEvidenceFile(p, "b-prkey", 1); + + contributeGithubEvidence(p, "w1", "org/r", { + prsSearched: true, + prsInWindow: [{ title: "first" }, { title: "second" }, { title: "third" }], + }, 2); + const got = readEvidenceFile(p).github["org/r"].prsInWindow; + assert.equal(got.length, 3, "three distinct unnumbered PRs must all survive"); + assert.deepEqual(got.map((x) => x.title), ["first", "second", "third"]); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("numbered PRs still merge across writers, string or numeric", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-prnum-")); + const { evidencePathFor, initEvidenceFile, contributeGithubEvidence, readEvidenceFile } = + await import("../lib/evidence-file.mjs"); + const p = evidencePathFor("b-prnum", dir); + initEvidenceFile(p, "b-prnum", 1); + + contributeGithubEvidence(p, "w1", "org/r", { prsInWindow: [{ pr: "#10", title: "a" }] }, 2); + contributeGithubEvidence(p, "w2", "org/r", { prsInWindow: [{ pr: 10, title: "a-updated" }] }, 3); + + const got = readEvidenceFile(p).github["org/r"].prsInWindow; + assert.equal(got.length, 1, "'#10' and 10 are the same PR"); + assert.equal(got[0].title, "a-updated", "later writer wins"); + + rmSync(dir, { recursive: true, force: true }); +}); From deac2718695e4e1242c67e8b4073752640c92643 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 16:05:57 +0530 Subject: [PATCH 33/51] =?UTF-8?q?fix(rca):=20make=20clustering=20persist?= =?UTF-8?q?=20=E2=80=94=20clusterRows=20silently=20discarded=20cluster=5Fi?= =?UTF-8?q?d?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the vrt_742 run's 30-minute wall clock, found by reading its transcript rather than inferring it. clusterRows() assigns cluster_id by MUTATING its input and returns {rows, clusters}. The orchestrator ran: const { clusters } = clusterRows(rows); // never writeRows(csvPath, rows) so the clusters printed correctly, the run looked clustered, and every cluster_id went to the CSV empty. Fan-out then had no representative/sibling structure and dispatched one coordinator per test: 12 tests became 26 subagents and 30.5 minutes, even though per-agent cost was healthy at 12.2 calls (vs a 45.4 baseline). The savings were real but spent on agent count instead of banked as time. Two independent callers made this exact mistake on the same day — including me, in the session that found it. That makes it an API footgun, not a user error, so the fix is at the API: clusterAndPersist(csvPath, csvState) reads, clusters, writes back, and throws if the persisted count doesn't match the row count. Step 3 now mandates it and tells the reader to verify cluster_id is non-empty before fan-out, since an unclustered run degrades silently to O(tests). 159 tests pass. Not pushed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/signature.mjs | 30 ++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 20 ++++++++++++++++++++ tests/signature.test.mjs | 29 +++++++++++++++++++++++++++++ 3 files changed, 79 insertions(+) diff --git a/lib/signature.mjs b/lib/signature.mjs index 42dc0ae..b2afe0e 100644 --- a/lib/signature.mjs +++ b/lib/signature.mjs @@ -76,3 +76,33 @@ export function clusterRows(rows) { return { rows, clusters }; } + +/** + * Seed-free, persist-safe clustering: read the CSV, assign `cluster_id`, and + * WRITE IT BACK. Returns the cluster objects. + * + * `clusterRows` mutates its input and returns `{rows, clusters}`, so a caller + * that destructures only `clusters` gets working cluster objects while every + * `cluster_id` is silently discarded — the CSV keeps empty cluster columns and + * the run degrades to one coordinator per test, losing the entire + * representative/sibling collapse. That is not a hypothetical: it happened on + * a real run (12 tests → 26 subagents, 30 minutes), and again to a second + * caller the same day. Two independent callers making the same mistake is an + * API problem, not a user problem. + * + * Prefer this over calling `clusterRows` directly whenever the rows came from + * a CSV. It cannot forget to persist. + */ +export function clusterAndPersist(csvPath, csvState) { + const { readRows, writeRows } = csvState; + const rows = readRows(csvPath); + const { clusters } = clusterRows(rows); + writeRows(csvPath, rows); + const persisted = readRows(csvPath).filter((r) => r.cluster_id).length; + if (rows.length && persisted !== rows.length) { + throw new Error( + `[signature] clustering wrote ${persisted}/${rows.length} cluster_id values — refusing to continue with a partially clustered CSV`, + ); + } + return clusters; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index ced942a..35cf693 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -240,6 +240,26 @@ write an empty CSV, report "no failed tests", stop. ## Step 3 — failure-signature clustering (see references/clustering.md) +Use **`clusterAndPersist(csvPath, csvStateModule)`** (`lib/signature.mjs`), not +`clusterRows` directly: + +```js +const clusters = clusterAndPersist(csvPath, await import("./lib/csv-state.mjs")); +``` + +`clusterRows` assigns `cluster_id` **in place** and returns `{rows, clusters}`, +so `const { clusters } = clusterRows(rows)` gives you working cluster objects +while every `cluster_id` is silently discarded — the CSV keeps empty cluster +columns and the run degrades to **one coordinator per test**, losing the whole +representative/sibling collapse. That is measured, not theoretical: a real run +went 12 tests → 26 subagents and 30 minutes with the clustering "done" but +never written. `clusterAndPersist` writes back and verifies the count, so it +cannot forget. + +Then verify before fan-out: **if `cluster_id` is empty on any row, Step 3 did +not take effect** — do not proceed, the run would silently cost O(tests) +instead of O(causes). + Compute a failure signature per row and assign `cluster_id` (`lib/signature.mjs`). Each cluster gets one **representative** (full multi-turn loop) and `N−1` **siblings** (pre-seeded one-turn confirm against their own logs). This collapses diff --git a/tests/signature.test.mjs b/tests/signature.test.mjs index f721167..c696b93 100644 --- a/tests/signature.test.mjs +++ b/tests/signature.test.mjs @@ -1,5 +1,8 @@ import { test } from "node:test"; import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { normalize, computeSignature, @@ -77,3 +80,29 @@ test("clusterRows stamps cluster_id onto every row", () => { assert.ok(rows.every((r) => r.cluster_id)); assert.notEqual(rows[0].cluster_id, rows[1].cluster_id); }); + +// clusterRows mutates its input; a caller that destructures only `clusters` +// silently loses every cluster_id. Two independent callers did exactly that on +// the same day, collapsing a clustered run into one coordinator per test. +test("clusterAndPersist writes cluster_id back to the CSV", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-cap-")); + const csvState = await import("../lib/csv-state.mjs"); + const { clusterAndPersist } = await import("../lib/signature.mjs"); + const csv = join(dir, "s.csv"); + csvState.seed(csv, "b", [ + { test_id: 1, test_name: "a", failure: { error_summary: "boom" } }, + { test_id: 2, test_name: "b", failure: { error_summary: "boom" } }, + { test_id: 3, test_name: "c", failure: { error_summary: "other" } }, + ]); + + const clusters = clusterAndPersist(csv, csvState); + assert.equal(clusters.length, 2, "two distinct signatures"); + + // The whole point: re-READ from disk, don't trust the in-memory rows. + const reread = csvState.readRows(csv); + assert.ok(reread.every((r) => r.cluster_id), "every row must have a persisted cluster_id"); + assert.equal(reread[0].cluster_id, reread[1].cluster_id, "same signature → same cluster"); + assert.notEqual(reread[0].cluster_id, reread[2].cluster_id); + + rmSync(dir, { recursive: true, force: true }); +}); From ab9ce220173b57de5b1e74ea4494fe660a29fc52 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 16:16:53 +0530 Subject: [PATCH 34/51] fix(rca): order siblings after their representative and require a pre_seed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sibling path DID run on the vrt_742 build — and inverted. Siblings averaged 22.7 tool calls and 2.2 turns against 8.0 and 2.0 for the representative they were meant to be a cheap fraction of; one burned 60 calls over 17 minutes. Cause: the coordinator has always accepted a `pre_seed` input, but Step 5 never said siblings must be dispatched AFTER their representative lands, and never said the dispatch prompt must carry the rep's finding. A "one-turn confirm" with nothing to confirm degenerates into a full independent investigation with the sibling framing on top — hence costing more, not less. Nothing detected it because the RCAs were still correct; only the cost was wrong. siblingPreSeed(csvPath, csvState, clusterId, representativeId) builds the seed from the representative's landed row and returns {ok:false, reason} when the rep is not resolved or recorded no root_cause — those siblings must not be dispatched yet. The seed carries an explicit "confirm against your own evidence, do not adopt this" instruction so the independence rule travels with it. Step 5 now states the per-cluster barrier: rep first, wait, then siblings. Clusters still run concurrently with each other; the barrier is per cluster. 160 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/signature.mjs | 52 +++++++++++++++++++++++++++++++++++++++ skills/rca-build/SKILL.md | 21 ++++++++++++++++ tests/signature.test.mjs | 33 +++++++++++++++++++++++++ 3 files changed, 106 insertions(+) diff --git a/lib/signature.mjs b/lib/signature.mjs index b2afe0e..0705835 100644 --- a/lib/signature.mjs +++ b/lib/signature.mjs @@ -106,3 +106,55 @@ export function clusterAndPersist(csvPath, csvState) { } return clusters; } + +/** + * Build the `pre_seed` a cluster sibling needs, from its representative's + * already-landed CSV row. Returns `{ok:false, reason}` if the representative + * is not terminal yet — meaning the sibling MUST NOT be dispatched. + * + * Siblings are only cheap because they confirm a hypothesis someone else + * already established. Dispatch one without that hypothesis and "one-turn + * confirm" degenerates into a full independent investigation — with the + * sibling framing on top, so it costs MORE than the representative it was + * meant to be a fraction of. Measured on a real run: siblings averaged 22.7 + * tool calls and 2.2 turns against 8.0 and 2.0 for the representative, and + * one burned 60 calls over 17 minutes. Nothing in the fan-out ordered them + * after their rep, and nothing refused to dispatch without a seed, so the + * degradation was silent. + * + * Fan-out contract: for each cluster, dispatch the representative, WAIT for it + * to land terminal, then dispatch its siblings with this seed. Clusters are + * independent, so they still run concurrently with each other. + */ +export function siblingPreSeed(csvPath, csvState, clusterId, representativeId) { + const rows = csvState.readRows(csvPath); + const rep = rows.find((r) => String(r.testRunId) === String(representativeId)); + if (!rep) return { ok: false, reason: `representative ${representativeId} not in the CSV` }; + + const state = String(rep.rca_done ?? "").toLowerCase(); + if (state !== "resolved") { + return { + ok: false, + reason: `representative ${representativeId} is "${rep.rca_done || "pending"}", not resolved — dispatching siblings now would make each one re-investigate from scratch`, + }; + } + if (!String(rep.root_cause ?? "").trim()) { + return { ok: false, reason: `representative ${representativeId} resolved but recorded no root_cause — nothing for a sibling to confirm` }; + } + + return { + ok: true, + clusterId, + representativeId: String(representativeId), + pre_seed: { + cause: rep.root_cause, + failure_type: rep.failure_type || "", + related_prs: rep.related_prs || "", + confidence: rep.confidence || "", + // Stated so the sibling confirms against ITS OWN evidence rather than + // adopting the verdict — the independence rule in Operating Principle 0. + instruction: + "Confirm or refute this against YOUR OWN test's evidence in one turn. Do not adopt it because it is written here.", + }, + }; +} diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 35cf693..59da2b7 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -400,6 +400,27 @@ every dispatch (representative and sibling) must be told to read it first. ## Step 5 — fan-out (fully autonomous) +**ORDER MATTERS: representative first, siblings only after it lands.** For each +cluster, dispatch the representative, wait for its row to go terminal, then +dispatch its siblings carrying `pre_seed` from +`siblingPreSeed(csvPath, csvState, clusterId, representativeId)`. Clusters are +independent, so they still run concurrently *with each other* — the barrier is +per cluster, not global. + +A sibling is only cheap because it confirms a hypothesis someone else already +established. Dispatch one without that hypothesis and "one-turn confirm" +degenerates into a full independent investigation *with the sibling framing on +top*, so it costs MORE than the representative it was meant to be a fraction of. +Measured on a real run: siblings averaged **22.7 tool calls and 2.2 turns** +against **8.0 and 2.0** for the representative, and one burned **60 calls over +17 minutes**. Nothing ordered them after their rep and nothing refused to +dispatch without a seed, so it degraded silently. + +`siblingPreSeed` returns `{ok:false, reason}` when the representative is not +resolved or recorded no `root_cause` — **do not dispatch that sibling yet**. +Never hand-roll the seed: the guard is the only thing standing between a +clustered run and O(tests) cost. + Drive the cluster work-list, **`concurrency` (default 20) at a time**: representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL (claim → heartbeat → flip) so the run is resumable. diff --git a/tests/signature.test.mjs b/tests/signature.test.mjs index c696b93..02dbfd0 100644 --- a/tests/signature.test.mjs +++ b/tests/signature.test.mjs @@ -106,3 +106,36 @@ test("clusterAndPersist writes cluster_id back to the CSV", async () => { rmSync(dir, { recursive: true, force: true }); }); + +// Siblings are only cheap because they confirm someone else's hypothesis. +// Dispatched without one they re-investigate from scratch — measured at 22.7 +// calls vs 8.0 for the representative they were meant to be a fraction of. +test("siblingPreSeed refuses to seed from an unfinished representative", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-seed-")); + const csvState = await import("../lib/csv-state.mjs"); + const { siblingPreSeed } = await import("../lib/signature.mjs"); + const csv = join(dir, "s.csv"); + csvState.seed(csv, "b", [ + { test_id: 1, test_name: "rep", failure: { error_summary: "boom" } }, + { test_id: 2, test_name: "sib", failure: { error_summary: "boom" } }, + ]); + + const early = siblingPreSeed(csv, csvState, "c-1", 1); + assert.equal(early.ok, false, "rep is still pending — must block"); + assert.match(early.reason, /not resolved/); + + // Resolved but with no root_cause is equally useless to a sibling. + csvState.flip(csv, 1, { rca_done: "resolved" }, 1000); + const empty = siblingPreSeed(csv, csvState, "c-1", 1); + assert.equal(empty.ok, false); + assert.match(empty.reason, /no root_cause/); + + csvState.flip(csv, 1, { rca_done: "resolved", root_cause: "PR #42 broke seeding", failure_type: "PRODUCT_BUG" }, 2000); + const ok = siblingPreSeed(csv, csvState, "c-1", 1); + assert.equal(ok.ok, true); + assert.equal(ok.pre_seed.cause, "PR #42 broke seeding"); + assert.equal(ok.pre_seed.failure_type, "PRODUCT_BUG"); + assert.match(ok.pre_seed.instruction, /Do not adopt it/, "independence must travel with the seed"); + + rmSync(dir, { recursive: true, force: true }); +}); From 6932e23a9d266aa354c5b40ce3050c292eef869d Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 16:46:13 +0530 Subject: [PATCH 35/51] fix(rca): make the base evidence file announce that it is a partial view MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Telling coordinators in the prompt to use evidence-show was not enough. Measured on the vrt_741 run: 21 of 25 evidence-file reads were raw cat/grep/Read against the base path and only 4 went through the folded view. A raw read shows the orchestrator's base ONLY and silently hides every contribution shard — precisely the representative-to-sibling context the file exists to carry, and the objective we were about to report as done. Three different agents each cat-ed the same file and each saw a partial picture. The file now says so in the first bytes anyone sees. JSON has no comments, so three `_`-prefixed marker keys are stamped first on every base write: what is missing, the exact evidence-show command to run instead, and why. They are stripped on read, so nothing downstream sees them as data, and re-stamping is idempotent rather than accumulating. Also checked what prompted this: the tool cache is barely used now because the redundancy it was built for is gone. vrt_741 made 169 shell calls of which only 6 were cross-agent repeats (4%), against the original "46 exact gh repeats for 10 tests". The MCP repeats are all same-agent getTfaTurnResult polling, which the cache correctly refuses as stateful. Low cache usage is the evidence file working, not the cache failing. 162 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/evidence-file.mjs | 38 +++++++++++++++++++++++++++++-- tests/evidence-file.test.mjs | 43 ++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/lib/evidence-file.mjs b/lib/evidence-file.mjs index 676cda5..e7e42d5 100644 --- a/lib/evidence-file.mjs +++ b/lib/evidence-file.mjs @@ -134,10 +134,42 @@ export function emptyEvidenceFile(buildId, nowMs) { /** The BASE file alone, no shards folded in. Internal to the orchestrator's * write path: `set*`/`merge*` must read-modify-write base only, or they would * absorb shard content into base and duplicate it on the next fold. */ +// Marker keys stamped on the BASE file so a raw read announces its own +// incompleteness. +// +// Telling coordinators in the prompt to use `evidence-show` was not enough: +// measured on a real run, 21 of 25 evidence-file reads were raw `cat`/`grep`/ +// `Read` against the base path, and only 4 went through the folded view. A raw +// read shows the orchestrator's base ONLY and silently hides every +// contribution shard — which is precisely the representative-to-sibling +// context the file exists to carry. Three different agents each `cat`-ed the +// same file and each saw a partial picture. +// +// So the file now says so itself, in the first bytes anyone sees. JSON has no +// comments, and these keys are the closest thing: they sort first, they are +// unmissable in a `cat` or a `head`, and they name the exact command to run. +// `_` prefixed and stripped on read, so they never reach the fold logic. +const BASE_MARKERS = { + _READ_ME_FIRST: + "PARTIAL VIEW — this is the orchestrator's BASE file only. Every coordinator's " + + "contribution lives in a separate shard alongside it and is NOT in this file. " + + "Reading this path directly (cat/grep/Read) WILL miss evidence other agents already gathered.", + _USE_INSTEAD: "node <pluginRoot>/bin/evidence-show.mjs <thisPath> --summary (also --prs, --repo <org/repo>)", + _WHY: "Only evidence-show folds base + all shards into the real view. A raw read has cost a real run duplicated work.", +}; + +/** Strip the marker keys — they are documentation for humans and agents, never + * data. Applied on every read so nothing downstream has to know about them. */ +function stripMarkers(doc) { + if (!doc || typeof doc !== "object") return doc; + for (const k of Object.keys(BASE_MARKERS)) delete doc[k]; + return doc; +} + export function readBaseFile(filePath) { if (!existsSync(filePath)) return emptyEvidenceFile("unknown-build", 0); try { - return JSON.parse(readFileSync(filePath, "utf8")); + return stripMarkers(JSON.parse(readFileSync(filePath, "utf8"))); } catch { return emptyEvidenceFile("unknown-build", 0); } @@ -236,7 +268,9 @@ export function writeEvidenceFile(filePath, doc) { // by every local user. Tighten an existing one too. if (dir) ensureOwnerOnlyDir(dir); const existed = existsSync(filePath); - writeFileSync(filePath, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); + // Markers first, so `head` and any truncated preview show them before data. + const stamped = { ...BASE_MARKERS, ...stripMarkers({ ...doc }) }; + writeFileSync(filePath, JSON.stringify(stamped, null, 2), { encoding: "utf8", mode: 0o600 }); // `mode` is only honoured when the file is CREATED. A file left over from a // run that predates this hardening would otherwise keep its old 0644 // forever, so tighten it explicitly on overwrite too. diff --git a/tests/evidence-file.test.mjs b/tests/evidence-file.test.mjs index db2bf8f..b9a84d8 100644 --- a/tests/evidence-file.test.mjs +++ b/tests/evidence-file.test.mjs @@ -405,3 +405,46 @@ test("numbered PRs still merge across writers, string or numeric", async () => { rmSync(dir, { recursive: true, force: true }); }); + +// Prompting agents to use evidence-show wasn't enough: 21 of 25 reads on a real +// run were raw cat/grep/Read against the base path, each silently missing every +// contribution shard. The file now announces that in its own first bytes. +test("a raw read of the base file announces that it is partial", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-warn-")); + const { evidencePathFor, initEvidenceFile, setGithubEvidence, readEvidenceFile, readBaseFile } = + await import("../lib/evidence-file.mjs"); + const { readFileSync } = await import("node:fs"); + const p = evidencePathFor("b-warn", dir); + initEvidenceFile(p, "b-warn", 1); + setGithubEvidence(p, "org/r", { deployState: { sha: "abc1234" } }, 2); + + const raw = readFileSync(p, "utf8"); + const head = raw.slice(0, 400); + assert.match(head, /PARTIAL VIEW/, "warning must be in the first bytes a cat/head shows"); + assert.match(raw, /evidence-show\.mjs/, "must name the command that gives the real view"); + + // Markers are documentation, never data — nothing downstream should see them. + for (const doc of [readBaseFile(p), readEvidenceFile(p)]) { + assert.equal(doc._READ_ME_FIRST, undefined); + assert.equal(doc._USE_INSTEAD, undefined); + assert.equal(doc._WHY, undefined); + } + // And the real content still round-trips. + assert.equal(readEvidenceFile(p).github["org/r"].deployState.sha, "abc1234"); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("markers survive repeated writes without accumulating", async () => { + const dir = mkdtempSync(join(tmpdir(), "rca-warn2-")); + const { evidencePathFor, initEvidenceFile, setGithubEvidence } = await import("../lib/evidence-file.mjs"); + const { readFileSync } = await import("node:fs"); + const p = evidencePathFor("b-w2", dir); + initEvidenceFile(p, "b-w2", 1); + for (let i = 0; i < 3; i++) setGithubEvidence(p, `org/r${i}`, { deployState: { sha: "abc1234" } }, i + 2); + + const raw = readFileSync(p, "utf8"); + assert.equal(raw.split("_READ_ME_FIRST").length - 1, 1, "exactly one marker, not one per write"); + + rmSync(dir, { recursive: true, force: true }); +}); From bc9cf65e73eda946cb96816a88c3e3b2313d9d3b Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 16:59:43 +0530 Subject: [PATCH 36/51] fix(rca): guard un-resumable pending-resume rows; document turnId/viewRca semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigated two things I had reported as open bugs. Both were wrong, and the investigation surfaced a real one underneath. NOT A BUG — turnId empty on resolved rows. TFA returns turnId ONLY on a soft- PENDING turn ({status, threadId, turnId}) and omits it on RESOLVED/NEEDS_INFO. Verified across two runs: 8 of 41 and 12 of 38 tool results carried a turnId, all of them PENDING. An empty turnId on a resolved row is correct. NOT A BUG — viewRca being a generic hostname. TFA itself returns "https://automation.browserstack.com — open the build's AI report (...)"; coordinators passed it through faithfully. The real per-build URL comes from triggerRcaReport at Step 6, once per run, not per test. THE REAL GAP. Because turnId exists only on PENDING — exactly the case that produces pending-resume — a row can land resumable with no turnId, and then the resume path cannot drain the in-flight turn: it submits blind onto a thread that still has a turn running. Such a row is indistinguishable from a healthy one in the CSV. flip() now warns loudly, naming the testRunId and what to do. A warning rather than a rejection: losing the row entirely is worse than resuming imperfectly. Coordinator doc now states both semantics so the next agent doesn't re-derive them or invent a more specific link than the data supports. Also removed four stray log files left in the working tree. 163 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- agents/ai-tfa-coordinator.md | 16 +++++++++++++++- lib/csv-state.mjs | 19 +++++++++++++++++++ tests/csv-state.test.mjs | 29 +++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 1 deletion(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index be2e006..97a7886 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -209,7 +209,21 @@ read-only and has no side effects, so a read is always safe to repeat. - `TFA agent run failed` — the wedge. Unrelated to message size (a 240-char message wedged like a 1500-char one). Fix: resubmit on the same thread, per 4b. - - `turn expired or not found` — observed on an over-cap (~2000-char) + - **`turnId` exists ONLY on a soft-`PENDING` turn.** TFA returns +`{status, threadId, turnId}` for PENDING and omits `turnId` entirely on +`RESOLVED` / `NEEDS_INFO` — so reporting `turn_id: not available` on a resolved +turn is correct, not a gap. What matters: if you end the test +`pending-resume`, you MUST carry the `turnId` from the PENDING response into +`flip()`, because the resume path drains that exact turn with +`getTfaTurnResult(testRunId, turnId)` before submitting anything new. Without +it the resume submits blind onto a thread that still has a turn in flight. + +**`viewRca` comes back from TFA as a generic hostname**, not a per-build deep +link. Pass through whatever TFA returns; do NOT hand-build a link that looks +more specific than the data supports. The real per-build report URL is produced +once at the end of the run by `triggerRcaReport`, not per test. + +`turn expired or not found` — observed on an over-cap (~2000-char) submit. The text names a thread/turn problem, which reads as a wedge and sends you down the wrong path; it is really a size rejection. If you see this, shorten and resend before assuming the thread is broken. diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 69c27e8..39a982b 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -331,6 +331,25 @@ export function flip(csvPath, testRunId, fields, nowMs) { } row.in_flight_worker = ""; row.timestamp = String(nowMs); + + // A `pending-resume` row is a PROMISE that this thread can be picked up + // again, and the resume path keeps that promise by calling + // getTfaTurnResult(testRunId, turnId) BEFORE submitting anything new. TFA + // returns a `turnId` only on a soft-`PENDING` turn — precisely the case that + // produces this state — so a resumable row without one cannot be drained: the + // resume would submit blind on a thread that still has an in-flight turn. + // + // Not an error, because losing the row entirely would be worse than resuming + // imperfectly. But it must be loud: silently un-resumable rows look identical + // to healthy ones in the CSV. + if (state === RESUMABLE && !String(row.turnId ?? "").trim()) { + console.warn( + `[csv-state] testRunId=${testRunId} flipped to ${RESUMABLE} with NO turnId — ` + + `resume cannot drain the in-flight turn and will submit blind. ` + + `Capture turnId from the PENDING tfaRcaTurn response and pass it to flip().`, + ); + } + writeRows(csvPath, rows); return true; } diff --git a/tests/csv-state.test.mjs b/tests/csv-state.test.mjs index 0dad786..1d7b3e3 100644 --- a/tests/csv-state.test.mjs +++ b/tests/csv-state.test.mjs @@ -255,3 +255,32 @@ test("writeRows tightens a pre-existing world-readable state dir", () => { rmSync(dir, { recursive: true, force: true }); }); + +// turnId only exists on a soft-PENDING turn, which is exactly the case that +// produces pending-resume. Without it the resume path submits blind onto a +// thread that still has a turn in flight — and the row looks healthy in the CSV. +test("flipping to pending-resume without a turnId warns loudly", () => { + const dir = mkdtempSync(join(tmpdir(), "rca-resume-")); + const csv = join(dir, "s.csv"); + seed(csv, "b", [{ test_id: 1, test_name: "t" }, { test_id: 2, test_name: "u" }]); + + const warnings = []; + const orig = console.warn; + console.warn = (m) => warnings.push(String(m)); + try { + flip(csv, 1, { rca_done: "pending-resume" }, 1000); + flip(csv, 2, { rca_done: "pending-resume", turnId: "abc-123" }, 1000); + } finally { + console.warn = orig; + } + + const noTurn = warnings.filter((w) => /NO turnId/.test(w)); + assert.equal(noTurn.length, 1, "exactly the seedless row must warn"); + assert.match(noTurn[0], /submit blind/); + assert.equal(readRows(csv).find((r) => r.testRunId === "2").turnId, "abc-123"); + + // Still resumable either way — warning, not rejection. + assert.equal(readRows(csv).find((r) => r.testRunId === "1").rca_done, "pending-resume"); + + rmSync(dir, { recursive: true, force: true }); +}); From 8d7adcac7f45e311de674aafca9a8afce058ce7b Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 18:03:31 +0530 Subject: [PATCH 37/51] docs(rca): add a generic API reference, and a test that keeps it from drifting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last two runs got slower and the cause was self-inflicted. On vrt_740, 92 of 407 tool calls (23%, 5.1 per test) were agents re-deriving the plugin's own API at runtime — `grep -n "^export function" lib/…`, `cat config/…`, repeated `ls .claude/skills/`. The previous run spent 13 (5%, 0.9 per test). That is ~79 of the ~120 extra calls between the runs, and most of the wall-clock regression from 18.2 to 36.3 minutes. Nothing was wrong with the helpers. Five were added this session to fix five silent correctness bugs, each justified — but the SKILL named them without signatures, so the only way for an agent to learn a call shape was to read the source. The plugin was taxing every agent to learn itself. Adds one "API reference" section before Step 0 with every signature a run needs, grouped by module, plus the bin/ commands, the three load-bearing constants (COLUMNS, RESUMABLE, TEST_LOGS) and the config keys. Fully generic: build ids, repos, branches, workloads and paths are all inputs supplied by the gate and the connector skills — verified no product or machine literals. Documenting it fixes today; drift is what actually caused this, so a test now asserts every exported lib helper appears in that section, with an explicit INTERNAL allowlist for module internals agents reach through bin/ instead. Verified it fires: adding an undocumented export fails the suite by name. For the record, this run also confirmed the fixes that preceded it: the base file's partial-view markers moved evidence reads from 16% to 74% folded (4:21 to 43:15), clustering persisted 18/18, the sibling barrier held with 0 duplicate dispatches, and gh calls per test fell 4.4 to 4.1 — so the hygiene fix displaced live gathering rather than adding to it. 164 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- skills/rca-build/SKILL.md | 79 +++++++++++++++++++++++++++++++++++++++ tests/wiring.test.mjs | 41 ++++++++++++++++++++ 2 files changed, 120 insertions(+) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 59da2b7..65a54b1 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -24,6 +24,85 @@ anything again.** Config (concurrency, turn-cap, paths, evidence registry) lives in `config/rca.config.json`. State lives in the CSV/WAL spine (`lib/csv-state.mjs`). +## API reference — read THIS, do not grep the source + +Every signature this run needs, in one place. Measured reason it exists: on one +run **92 of 407 tool calls (23%, 5.1 per test)** were agents re-deriving this — +`grep -n "^export function" lib/…`, `cat config/…`, repeated `ls .claude/skills/`. +The previous run spent 13. The jump came from adding helpers faster than the +docs described them, so the plugin taxed every agent to learn itself. + +Everything below is product-neutral: build ids, repos, branches, workloads and +paths are all **inputs**, supplied by the gate and the connector skills. + +**State spine — `lib/csv-state.mjs`** +``` +csvPathFor(buildId, stateDir="") → <stateDir|tmpdir>/bstack-rca/rca-state.<buildId>.csv +seed(csvPath, buildId, tests) → rows; idempotent, preserves terminal rows +readRows(csvPath) / writeRows(csvPath,rows) throws on a foreign header rather than dropping columns +claim(csvPath, testRunId, worker, nowMs) → false if already claimed +heartbeat(csvPath, testRunId, worker, nowMs) +flip(csvPath, testRunId, fields, nowMs) → false if rca_done missing/non-terminal +reaper(csvPath, ttlSec, nowMs) → reclaimed ids +pendingRows(csvPath) → pending + pending-resume +``` + +**Clustering — `lib/signature.mjs`** +``` +clusterAndPersist(csvPath, csvStateModule) → clusters; WRITES cluster_id back. Use this. +siblingPreSeed(csvPath, csvState, clusterId, repId) → {ok, pre_seed} | {ok:false, reason} +clusterRows(rows) → {rows, clusters}; mutates, does NOT persist +``` + +**Shared evidence — `lib/evidence-file.mjs`** +``` +evidencePathFor(buildId, stateDir="") initEvidenceFile(path, buildId, nowMs) +setGithubEvidence(path, repo, entry, nowMs) setLogsEvidence(path, workload, entry, nowMs) +setBaseline(path, baseline, suspectWindow, nowMs) setLocalRepos(path, localRepos, nowMs) +contributeGithubEvidence(path, writerId, repo, patch, nowMs) ← coordinators write HERE +contributeLogsEvidence(path, writerId, workload, patch, nowMs) +deployShas(pathOrDoc) → {pins:{repo:sha}, source} recomputeCoverage(path, {repos,workloads}, nowMs) +readEvidenceFile(path) folds base+shards · readBaseFile(path) is base ONLY +``` + +**Local repo reads — `lib/repo-source.mjs`** +``` +discoverWorkspaceRoot({repos, from, explicit, maxTries=3}) → {root, matched, tried, reason} +resolveLocalRepos({repos, pins, workspaceRoot}) → {repo:{usable, sha|reason}} +readFileAt({repo, sha, path, workspaceRoot}) → sha ONLY; a branch name is refused +``` + +**Housekeeping — `lib/state-dir.mjs`** +``` +hardenStateDir(dir) run once at gate start; idempotent +pruneStateDir(dir, nowMs, {maxAgeMs, dryRun}) NOT automatic — these files are the resume state +``` + +**Routing / output — `lib/routing.mjs`, `lib/glimpse.mjs`, `lib/evidence-cache.mjs`** +``` +loadConfig(configPath) buildManifest(config, discovered) routeAsks(asks, config, manifest) +renderGlimpseFromCsv(csvPath, {buildId}) resolveBaseline(lastGreenRef, fallbackRef) +``` + +**Commands — `bin/`** +``` +node bin/evidence-show.mjs <evidenceFile> [--summary | --prs | --repo <org/repo>] +node bin/repo-read.mjs <buildId> <writerId> <org/repo> <sha> <path> [--fetch] +node bin/cached-exec.mjs <buildId> <writerId> '<command>' (pipe OUTSIDE the wrapper) +node bin/cached-mcp.mjs <buildId> get|put <tool> '<argsJson>' +``` + +**Constants worth knowing** +``` +csv-state.COLUMNS the canonical column set; writeRows emits exactly these +csv-state.RESUMABLE "pending-resume" — a SOFT terminal: claim released, row still picked up +routing.TEST_LOGS the ask type TFA owns; never gather it, always skip +``` + +**Config** — `config/rca.config.json`: `concurrency`, `turnCap`, `softPendingDrain`, +`reaperHeartbeatTtlSec`, `paths.stateDir`, `evidenceRouting`. Read it once at the +gate and pass the values down; a coordinator should never need to open it. + ## Step 0 — input Parse the build id from the invocation args. Accepted forms: a bare build id, a diff --git a/tests/wiring.test.mjs b/tests/wiring.test.mjs index 2b7c004..4138bf6 100644 --- a/tests/wiring.test.mjs +++ b/tests/wiring.test.mjs @@ -74,3 +74,44 @@ test("gate-critical lib exports are actually invoked outside tests", () => { assert.ok(uses >= 2, `${fn} appears ${uses}x outside tests — defined but never driven`); } }); + +// The root cause of the 23% discovery tax was DRIFT: helpers were added faster +// than the docs described them, so agents grepped lib/ at runtime to learn the +// API. Documenting it once fixes today; this test keeps it fixed. +test("every exported lib helper appears in the SKILL's API reference", () => { + const skill = readFileSync(join(ROOT, "skills/rca-build/SKILL.md"), "utf8"); + + // Internal-by-convention: replay/test seams and trivial helpers a coordinator + // never calls. Anything NOT listed here must be documented. + const INTERNAL = new Set([ + "emptyEvidenceFile", "writeEvidenceFile", "contribDirFor", "contribPathFor", + "hasTrustworthyPrList", "stalenessOf", "makeEvidenceCache", + "replaySubmit", "replayRead", "normalize", "computeSignature", + "selectRepresentative", "localCloneFor", "hasCommit", "ensureCommit", + "classifyCoverage", "coverageStamp", "orderAsks", "routeAsk", + "unavailableCapabilities", "renderGlimpse", "toolCacheDirFor", "cacheKey", + "isCacheable", "splitPipeline", + // tool-cache module internals — agents drive the cache through + // bin/cached-exec.mjs / bin/cached-mcp.mjs, never by importing it. + "isRunnable", "tokenize", "isCacheableMcp", "redact", "cacheGet", + "cachePut", "cacheStats", "mcpCacheKey", + ]); + + const undocumented = []; + for (const f of readdirSync(join(ROOT, "lib")).filter((f) => f.endsWith(".mjs"))) { + const src = readFileSync(join(ROOT, "lib", f), "utf8"); + for (const m of src.matchAll(/^export (?:function|const) ([A-Za-z0-9_]+)/gm)) { + const name = m[1]; + if (INTERNAL.has(name)) continue; + if (!skill.includes(name)) undocumented.push(`${f}:${name}`); + } + } + + assert.deepEqual( + undocumented, + [], + `undocumented helper(s): ${undocumented.join(", ")}. Add them to the SKILL's ` + + `"API reference" section — an agent that can't find a signature there greps ` + + `lib/ at runtime, which cost 92 of 407 tool calls on one measured run.`, + ); +}); From 9d24c86cfa3ac3265a2afd1757b6ba581e0e0a21 Mon Sep 17 00:00:00 2001 From: harshit-bstack <harshit.m@browserstack.com> Date: Sun, 2 Aug 2026 18:10:26 +0530 Subject: [PATCH 38/51] fix(rca): enforce the product-bug evidence rule instead of only stating it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swept for the two bug classes this session kept producing — mutate-and-return footguns, and hard rules that live only in the prompt. Class 1 (mutate-and-return) came back clean: clusterRows is the only one, and clusterAndPersist already covers it. A pendingRows hit was a false positive in my own grep, not a defect. Class 2 turned up "A PRODUCT_BUG RCA without a culprit PR is incomplete" — load-bearing, and enforced by nothing. Checked it against four real builds before changing anything: it has held. Of 9 product-bug rows, 3 carried a PR link and 6 carried an explicit "none — searched <what>", which the rule permits. Zero were blank. So this is a guard, not a bug fix. Worth adding anyway: a blank related_prs is indistinguishable in the CSV from a genuine dead end, so the failure would be a silent product-bug attribution with no evidence trail — the same shape as every other bug found this session. flip() now warns when failure_type is a product bug and related_prs is EMPTY. A stated "none, searched X" is compliant and deliberately not nagged. 165 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- lib/csv-state.mjs | 17 +++++++++++++++++ tests/csv-state.test.mjs | 29 +++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/lib/csv-state.mjs b/lib/csv-state.mjs index 39a982b..5d636f5 100644 --- a/lib/csv-state.mjs +++ b/lib/csv-state.mjs @@ -342,6 +342,23 @@ export function flip(csvPath, testRunId, fields, nowMs) { // Not an error, because losing the row entirely would be worse than resuming // imperfectly. But it must be loud: silently un-resumable rows look identical // to healthy ones in the CSV. + // "A PRODUCT_BUG RCA without a culprit PR is incomplete" is one of the + // plugin's hard rules, and it lived only in the prompt. It has held so far — + // across four builds, 3 such rows carried a PR link and 6 carried an explicit + // "none — searched <what>", which the rule permits. Zero were blank. But an + // unenforced rule is one distracted turn away from a silent product-bug + // attribution with no evidence trail, and a blank field is indistinguishable + // from a genuine dead end in the CSV. Cheap to guard, so guard it. + // + // A stated "none, searched X" satisfies the rule; only EMPTY does not. + if (/PRODUCT_BUG|application/i.test(String(row.failure_type ?? "")) && + !String(row.related_prs ?? "").trim()) { + console.warn( + `[csv-state] testRunId=${testRunId} is ${row.failure_type} with an EMPTY related_prs — ` + + `record the culprit PR link, or state what was searched and why none was found.`, + ); + } + if (state === RESUMABLE && !String(row.turnId ?? "").trim()) { console.warn( `[csv-state] testRunId=${testRunId} flipped to ${RESUMABLE} with NO turnId — ` + diff --git a/tests/csv-state.test.mjs b/tests/csv-state.test.mjs index 1d7b3e3..6070ef4 100644 --- a/tests/csv-state.test.mjs +++ b/tests/csv-state.test.mjs @@ -284,3 +284,32 @@ test("flipping to pending-resume without a turnId warns loudly", () => { rmSync(dir, { recursive: true, force: true }); }); + +// "A PRODUCT_BUG RCA without a culprit PR is incomplete" was a prompt-only rule. +// A stated "none — searched X" satisfies it; a blank field does not, and the two +// are indistinguishable in the CSV. +test("flip warns on a product bug with no PR evidence trail", () => { + const dir = mkdtempSync(join(tmpdir(), "rca-pb-")); + const csv = join(dir, "s.csv"); + seed(csv, "b", [{ test_id: 1 }, { test_id: 2 }, { test_id: 3 }]); + + const warnings = []; + const orig = console.warn; + console.warn = (m) => warnings.push(String(m)); + try { + flip(csv, 1, { rca_done: "resolved", failure_type: "PRODUCT_BUG" }, 1); + flip(csv, 2, { rca_done: "resolved", failure_type: "PRODUCT_BUG", related_prs: ["https://x/pull/1"] }, 1); + flip(csv, 3, { rca_done: "resolved", failure_type: "PRODUCT_BUG", related_prs: "none — searched repo-a, repo-b in window" }, 1); + } finally { + console.warn = orig; + } + + const pb = warnings.filter((w) => /EMPTY related_prs/.test(w)); + assert.equal(pb.length, 1, "only the blank one warns"); + assert.match(pb[0], /testRunId=1/); + + // An honest dead end is compliant — must not be nagged. + assert.ok(!pb.some((w) => /testRunId=3/.test(w)), "a stated 'none, searched X' satisfies the rule"); + + rmSync(dir, { recursive: true, force: true }); +}); From 1b13b07c43400d138027c26611772ff6dc63b250 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 00:33:25 +0530 Subject: [PATCH 39/51] feat(rca): pre-dispatch a cluster representative's turn 1 concurrently with Step 4, and gather NEEDS_INFO asks concurrently MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 4b submits each cluster representative's turn 1 directly (tfaRcaTurn), in the same tool-call batch as Step 4's evidence pre-fetch, instead of waiting for Step 5's coordinator dispatch to submit it. A RESOLVED turn 1 flips the CSV row straight to terminal and dispatches its siblings immediately, with no Step 5 dispatch at all for that representative; a NEEDS_INFO/PENDING outcome is handed to the eventual coordinator via the new turn1_result/resume inputs (lib/turn1-registry.mjs) so turn 1 is never resubmitted. Wired through all three execution paths: lib/loop.mjs (sequential harness), workflows/rca-batch.mjs (Workflow path, plus a resume/turn1_result mutual-exclusivity guard so a prior-run pending-resume thread always wins over a same-run Step 4b entry), and agents/ai-tfa-coordinator.md (direct-dispatch prose contract). Also parallelizes a NEEDS_INFO turn's independent gather() calls (lib/routing.mjs's routeAsk/routeAsks classify each ask with no cross-ask state — priority only orders the assembled message, never the fetching) via Promise.all instead of one round-trip at a time, mirrored in the coordinator's own prose contract. Coordinator paths (references/evidence-routing.md, references/github-evidence.md) are also now consistently pluginRoot-qualified rather than bare, so a dispatched coordinator never resolves them against the wrong cwd. --- agents/ai-tfa-coordinator.md | 47 +++++++-- lib/loop.mjs | 33 ++++-- lib/turn1-registry.mjs | 117 +++++++++++++++++++++ tests/loop-parallel-gather.test.mjs | 154 ++++++++++++++++++++++++++++ tests/loop-turn1-result.test.mjs | 110 ++++++++++++++++++++ tests/turn1-registry.test.mjs | 128 +++++++++++++++++++++++ workflows/rca-batch.mjs | 75 ++++++++++++-- 7 files changed, 643 insertions(+), 21 deletions(-) create mode 100644 lib/turn1-registry.mjs create mode 100644 tests/loop-parallel-gather.test.mjs create mode 100644 tests/loop-turn1-result.test.mjs create mode 100644 tests/turn1-registry.test.mjs diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 97a7886..d92e88c 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -31,12 +31,32 @@ it names no `kubectl` / `chitragupta` / `bifrost`; it routes by *capability*. ## Inputs +- `pluginRoot` — **required**, absolute path to this plugin's repo root. Every + `<pluginRoot>/...` path in this file (bin/ commands, reference docs, the API + reference in `skills/rca-build/SKILL.md`) is relative to this value, not to + whatever directory you were started in. Missing it is what causes a + coordinator to guess `references/<file>.md` against the wrong cwd and burn a + `find` recovering the real path — the dispatch prompt must state it up front. - `testRunId` — **required**, the integer test-run ID. Maps to the tool's `testRunId` arg. - `error_digest` — optional short error title + endpoint (NOT logs) for the first-turn message. - `pre_seed` — optional. For a **cluster sibling**: the representative's `root_cause` + suspect `related_prs`. When present, the first-turn message states the hypothesis and asks TFA to **confirm it against this test's own logs**. - `resume` — optional `{ threadId, turnId }` from a prior PENDING run. +- `turn1_result` — optional `{ threadId, asks }`. Set only for a cluster + representative whose turn 1 was already pre-submitted by the orchestrator's + Step 4b pass (`skills/rca-build/SKILL.md` Step 4b, `lib/turn1-registry.mjs`) + and landed `NEEDS_INFO` — i.e. a real answer already exists, just not a + terminal one. When present, **do not submit turn 1** — start the loop + already at step 3 (ROUTE the asks) using `turn1_result.asks`, with + `threadId = turn1_result.threadId` and `turns_used` starting at `1`. Mutually + exclusive with `resume` and `pre_seed` per dispatch: a representative gets at + most one of `resume` (Step 4b's turn 1 was still soft-`PENDING`), + `turn1_result` (Step 4b's turn 1 already resolved to `NEEDS_INFO`), or + neither (Step 4b never ran, e.g. an unclustered rerun) — never more than one, + and never alongside `pre_seed`, which is sibling-only. A Step 4b turn 1 that + landed `RESOLVED` needs no coordinator dispatch at all: the orchestrator + flips that row straight to terminal and this agent is never invoked for it. - `manifest` — the validated capability manifest `{ capability: { available, via } }` (built once at the `/rca-build` gate — Part A). - `evidenceFile` — optional. Absolute path to the build-level pre-fetch @@ -263,7 +283,9 @@ once at the end of the run by `triggerRcaReport`, not per test. before — never busy-wait through `tfaRcaTurn` resubmits instead. 6. **Digest, don't dump.** Every follow-up `message` carries digested findings (`ask → found → snippet/link`), never raw log tails, full diffs, or full files. - Size caps + block shape live in `references/evidence-routing.md` — read it + Size caps + block shape live in `<pluginRoot>/skills/rca-build/references/evidence-routing.md` + (NOT a bare `references/evidence-routing.md` — that resolves against + whatever directory you started in, not this plugin's root) — read it before fulfilling any ask. The plugin config caps `message` at 1000 chars (`turnMessageMaxChars` in `config/rca.config.json`); the `tfaRcaTurn` tool itself would allow up to 5000, but the plugin self-limits to 1000. @@ -284,7 +306,7 @@ once at the end of the run by `triggerRcaReport`, not per test. signature blocks, unrequested columns) than any evidence ask ever uses. This governs what enters *your own* context via the tool result — distinct from principle 6, which governs the digest you send back to TFA. Exact - command templates: `references/github-evidence.md` § Field-filtering. + command templates: `<pluginRoot>/skills/rca-build/references/github-evidence.md` § Field-filtering. ## Application bugs — the culprit-PR mandate (MANDATORY) @@ -293,7 +315,7 @@ Whenever TFA's classification (in an ask, a suggestion, or the resolving connector is the deliverable, not optional evidence: - **Hunt the culprit PR**: deploy timeline vs the last-pass window, changed - paths vs the failure signature (`references/github-evidence.md`), run the + paths vs the failure signature (`<pluginRoot>/skills/rca-build/references/github-evidence.md`), run the falsification protocol on each candidate. - **Feed the PR link(s) to TFA in the turn message** so the BrowserStack agent populates `related_prs` in the dashboard RCA. @@ -306,7 +328,7 @@ connector is the deliverable, not optional evidence: ## Suspect-PR falsification (github asks) -For `product_code` / `deploy` / `ci` asks, follow `references/github-evidence.md`: +For `product_code` / `deploy` / `ci` asks, follow `<pluginRoot>/skills/rca-build/references/github-evidence.md`: gather the **exact** evidence (diff-since-baseline, PRs-in-window touching the failing path, blame, deploy timing) via **GitHub MCP → `gh` → degrade**, and for each candidate suspect **try to disprove it** (path overlap? shipped before the @@ -332,6 +354,9 @@ capability is unavailable — emit an - neither → "Initiating collaborative RCA for test run <id>." 1. SUBMIT turn 1: tfaRcaTurn(testRunId=<id>, message=<digest>). Capture threadId. turns_used = 1. (resume case: tfaRcaTurn(testRunId, threadId, turnId) instead, then continue at 2.) + (turn1_result case: SKIP this submit entirely — threadId = turn1_result.threadId, + turns_used = 1, result.status = NEEDS_INFO, result.asks = turn1_result.asks, + then continue at 3, not 2 — there is nothing to CLASSIFY, Step 4b already did.) 2. CLASSIFY result.status: PENDING → DRAIN FIRST, do not resubmit and do not end here: capture threadId + turnId, then loop on @@ -344,8 +369,18 @@ capability is unavailable — emit an RESOLVED → capture glimpse + viewRca; END (RESOLVED). BLOCKED → END (PENDING, note "blocked") — terminal, no asks to route. NEEDS_INFO → go to 3. -3. ROUTE the asks (read references/evidence-routing.md; route via lib/routing.mjs): - For each ask, high → medium → low: +3. ROUTE the asks (read `<pluginRoot>/skills/rca-build/references/evidence-routing.md`; route via lib/routing.mjs): + "high → medium → low" orders the ASSEMBLED MESSAGE only (step 3's last + line) — `routeAsk`/`routeAsks` (`lib/routing.mjs`) classify each ask + independently, with no cross-ask state or ordering dependency between one + ask's gather and another's. When a turn's NEEDS_INFO carries multiple + `gather` asks (e.g. a github ask and an infra ask together), issue their + live gather calls CONCURRENTLY — as parallel tool calls in the same + turn — never one ask's full gather-and-digest before starting the next. + `lib/loop.mjs`'s `runRcaLoop` mirrors this with `Promise.all` over + `buckets.gather`; do the equivalent here. Only the final message assembly + respects priority order, not the fetching. + For each ask: skip → record in asks_skipped, emit nothing. gather → FIRST check `evidenceFile` (if present) for this ask's scope — repo for a github ask, workload for an infra/logs ask. Covered diff --git a/lib/loop.mjs b/lib/loop.mjs index d9535fa..e0ef66b 100644 --- a/lib/loop.mjs +++ b/lib/loop.mjs @@ -128,6 +128,15 @@ async function drainSoftPending({ testRunId, pending, readTurn, sleep, drain }) // submit({ testRunId, message, threadId, turnId }) → Promise<turn> (tfaRcaTurn shape) // readTurn({ testRunId, turnId }) → Promise<turn> (getTfaTurnResult shape) // gather(routedGatherEntry) → Promise<string> (one digest block) +// turn1Result: { threadId, asks } — a Step 4b pre-dispatch (SKILL.md Step 4b, +// lib/turn1-registry.mjs) already submitted turn 1 for this representative +// and it landed NEEDS_INFO. When present, turn 1 is NEVER submitted again — +// the loop starts already at the ROUTE step with this thread's asks, same +// as `agents/ai-tfa-coordinator.md`'s `turn1_result` input. A pre-dispatch +// that landed PENDING instead uses the existing `resume` convention +// (thread the drained turnId in via the caller's own resume handling) — +// it needs no special case here, since draining a soft-PENDING and then +// re-classifying is exactly what this loop already does. export async function runRcaLoop({ testRunId, firstMessage = "", @@ -139,6 +148,7 @@ export async function runRcaLoop({ turnCap = config?.turnCap ?? 6, drain = config?.softPendingDrain, sleep = defaultSleep, + turn1Result, }) { if (testRunId == null || Number.isNaN(Number(testRunId))) { return { @@ -181,7 +191,14 @@ export async function runRcaLoop({ while (true) { turns++; - let turn = await submit({ testRunId, message, threadId, turnId }); + // Step 4b already ran turn 1 for this representative and it landed + // NEEDS_INFO — treat it as this iteration's result instead of resubmitting + // message 1. Only applies on the very first pass; every later iteration + // submits normally regardless of what turn1Result held. + let turn = + turns === 1 && turn1Result?.threadId + ? { status: "NEEDS_INFO", threadId: turn1Result.threadId, asks: turn1Result.asks ?? [] } + : await submit({ testRunId, message, threadId, turnId }); threadId = turn.threadId ?? threadId; // Soft-PENDING → the in-call poll capped out, not a verdict. Read the SAME @@ -225,12 +242,16 @@ export async function runRcaLoop({ // Route + fulfill. Gaps degrade to `unavailable` — never a user prompt. const buckets = routeAsks(turn.asks ?? [], config, manifest); - const blocks = []; for (const s of buckets.skip) skipped.add(s.evidenceType); - for (const g of buckets.gather) { - blocks.push(await gather(g)); - fulfilled.add(g.evidenceType); - } + // Independent asks: routeAsk/routeAsks (lib/routing.mjs) do pure per-ask + // classification with no cross-ask state, and "high -> medium -> low" only + // orders the assembled message, never a data dependency between one ask's + // gather and another's. So fetch them concurrently instead of one + // round-trip at a time — Promise.all preserves buckets.gather's priority + // order in the result, so message order is unchanged. + const gathered = await Promise.all(buckets.gather.map((g) => gather(g))); + buckets.gather.forEach((g) => fulfilled.add(g.evidenceType)); + const blocks = [...gathered]; for (const gap of buckets.gap) { unavailable.add(gap.evidenceType); blocks.push(unavailableBlock(gap)); diff --git a/lib/turn1-registry.mjs b/lib/turn1-registry.mjs new file mode 100644 index 0000000..d0674c9 --- /dev/null +++ b/lib/turn1-registry.mjs @@ -0,0 +1,117 @@ +// Step 4b pre-dispatch registry (see skills/rca-build/SKILL.md Step 4b). +// +// Step 4b submits tfaRcaTurn's FIRST turn for every cluster representative +// directly from the orchestrator, in the same tool-call batch as Step 4's +// evidence pre-fetch — so the representative's turn 1 is already in flight +// (or already answered) by the time Step 5 would otherwise submit it fresh. +// +// A RESOLVED turn 1 needs no registry entry at all: the orchestrator flips +// that row straight to terminal in the CSV (lib/csv-state.mjs) and Step 5 +// skips dispatching a coordinator for it entirely. Only the two non-terminal +// outcomes are recorded here, for Step 5 to hand to the representative's +// coordinator instead of letting it submit turn 1 again: +// - PENDING -> {threadId, turnId} (drain via the existing `resume` input) +// - NEEDS_INFO -> {threadId, asks} (new `turn1_result` input — see +// agents/ai-tfa-coordinator.md and lib/loop.mjs) +// +// Single-writer: only the Step 4b orchestrator pass ever writes this file (one +// process, one point in time, before any coordinator is dispatched). Step 5 +// only reads it once per representative while building dispatch prompts, so — +// unlike the evidence file's per-coordinator shards — a plain read-modify-write +// is safe; there is no concurrent-writer race to design around here. +// +// Path convention mirrors csvPathFor/evidencePathFor exactly: build id in the +// filename, OS temp by default, `stateDir` overrides the directory only. + +import { readFileSync, writeFileSync, existsSync, mkdirSync, chmodSync, rmSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { tmpdir } from "node:os"; + +const safeName = (v, fallback) => + String(v ?? "").replace(/[^A-Za-z0-9._-]/g, "_") || fallback; + +export function turn1PathFor(buildId, stateDir = "") { + const safe = safeName(buildId, "unknown-build"); + const dir = stateDir && String(stateDir).trim() !== "" ? String(stateDir) : join(tmpdir(), "bstack-rca"); + return join(dir, `rca-turn1.${safe}.json`); +} + +// Owner-only, on create AND on an existing directory — same rationale as +// csv-state.mjs / evidence-file.mjs: a directory made before this hardening +// landed keeps its 0755 forever, and this file carries thread ids and NEEDS_INFO +// ask text in a shared OS temp dir. +function ensureOwnerOnlyDir(dir) { + if (!existsSync(dir)) { mkdirSync(dir, { recursive: true, mode: 0o700 }); return; } + try { chmodSync(dir, 0o700); } catch { /* not ours to tighten; leave it */ } +} + +function emptyRegistry(buildId, nowMs) { + return { buildId: String(buildId ?? ""), generatedAtMs: nowMs, entries: {} }; +} + +function readDoc(filePath) { + if (!existsSync(filePath)) return null; + try { + return JSON.parse(readFileSync(filePath, "utf8")); + } catch { + return null; + } +} + +function writeDoc(filePath, doc) { + const dir = dirname(filePath); + if (dir) ensureOwnerOnlyDir(dir); + const existed = existsSync(filePath); + writeFileSync(filePath, JSON.stringify(doc, null, 2), { encoding: "utf8", mode: 0o600 }); + // `mode` is only honoured on create — tighten a pre-hardening leftover too. + if (existed) chmodSync(filePath, 0o600); +} + +/** Idempotent: creates the file with the given `buildId` if it doesn't exist + * yet; leaves an existing file untouched otherwise (never clobbers prior + * entries on a resume). */ +export function initTurn1Registry(filePath, buildId, nowMs) { + const existing = readDoc(filePath); + if (existing) return existing; + const doc = emptyRegistry(buildId, nowMs); + writeDoc(filePath, doc); + return doc; +} + +/** + * Record a representative's non-terminal turn-1 outcome. + * `entry` shape: `{ threadId, turnId, status: "PENDING" | "NEEDS_INFO", asks, note }`. + * `turnId` only applies to PENDING (per the tfaRcaTurn contract — RESOLVED and + * NEEDS_INFO never carry one). `asks` only applies to NEEDS_INFO. + * Read-modify-write against the whole file — safe because Step 4b is this + * file's only writer. + */ +export function recordTurn1(filePath, testRunId, entry, nowMs) { + const doc = readDoc(filePath) ?? emptyRegistry("unknown-build", nowMs); + doc.entries[String(testRunId)] = { ...entry, submittedAtMs: nowMs }; + doc.generatedAtMs = nowMs; + writeDoc(filePath, doc); + return doc.entries[String(testRunId)]; +} + +/** This representative's pre-dispatched turn-1 outcome, or `null` if Step 4b + * never ran for it (not clustered as a representative, resolved already and + * flipped straight to the CSV, or the registry doesn't exist at all). */ +export function readTurn1(filePath, testRunId) { + const doc = readDoc(filePath); + return doc?.entries?.[String(testRunId)] ?? null; +} + +/** Every recorded entry, keyed by testRunId — used for run-end stats only. */ +export function readAllTurn1(filePath) { + return readDoc(filePath)?.entries ?? {}; +} + +/** Used by lib/build-cleanup.mjs once the build's report has generated + * successfully. A missing file is not an error — Step 4b may never have run + * (no clusters, or every representative resolved on turn 1). */ +export function deleteTurn1Registry(filePath) { + if (!existsSync(filePath)) return false; + rmSync(filePath, { force: true }); + return true; +} diff --git a/tests/loop-parallel-gather.test.mjs b/tests/loop-parallel-gather.test.mjs new file mode 100644 index 0000000..a05f56f --- /dev/null +++ b/tests/loop-parallel-gather.test.mjs @@ -0,0 +1,154 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { runRcaLoop } from "../lib/loop.mjs"; + +// A NEEDS_INFO turn's independent asks (lib/routing.mjs's routeAsk/routeAsks +// have no cross-ask state) should be gathered concurrently, not one round-trip +// at a time. This file proves both properties of that fix: the calls actually +// overlap, and the final message still assembles blocks in priority order +// regardless of which one finishes first. + +const CONFIG = { + turnCap: 6, + evidenceRouting: { + test_logs: { owner: "tfa", skip: true }, + product_code: { capability: "github" }, + infra: { capability: "infra" }, + other: { capability: "other" }, + }, +}; +const MANIFEST = { + github: { available: true, via: "gh" }, + infra: { available: true, via: "kubectl" }, +}; +const resolved = (threadId) => ({ + status: "RESOLVED", + threadId, + confidence: "high", + glimpse: { root_cause: "r", failure_type: "f", related_prs: [] }, + viewRca: "https://automation.browserstack.com/x", +}); +const delay = (ms) => new Promise((r) => setTimeout(r, ms)); + +test("independent NEEDS_INFO gathers run concurrently, not sequentially", async () => { + const events = []; + const gather = async (g) => { + events.push(`start:${g.evidenceType}`); + // The slower ask (product_code) is listed FIRST but finishes LAST. If + // gathers were sequential, infra's "start" could never appear before + // product_code's "end". + await delay(g.evidenceType === "product_code" ? 30 : 5); + events.push(`end:${g.evidenceType}`); + return `BLOCK:${g.evidenceType}`; + }; + + let calls = 0; + const submit = async () => { + calls++; + if (calls === 1) { + return { + status: "NEEDS_INFO", + threadId: "chat:1", + asks: [ + { evidenceType: "product_code", priority: "high", ask: { what: "diff" } }, + { evidenceType: "infra", priority: "medium", ask: { what: "pod status" } }, + ], + }; + } + return resolved("chat:1"); + }; + + await runRcaLoop({ + testRunId: "1", + firstMessage: "start", + submit, + config: CONFIG, + manifest: MANIFEST, + gather, + }); + + const firstEnd = events.findIndex((e) => e.startsWith("end:")); + const startsBeforeFirstEnd = events.slice(0, firstEnd).filter((e) => e.startsWith("start:")); + assert.equal( + startsBeforeFirstEnd.length, + 2, + `expected both gathers to start before either finished, got: ${events.join(", ")}`, + ); +}); + +test("gathered blocks preserve priority order in the message even when the slower ask finishes first", async () => { + const gather = async (g) => { + await delay(g.evidenceType === "product_code" ? 30 : 5); + return `BLOCK:${g.evidenceType}`; + }; + + let calls = 0; + const submits = []; + const submit = async (args) => { + calls++; + submits.push(args); + if (calls === 1) { + return { + status: "NEEDS_INFO", + threadId: "chat:2", + // Listed low-priority-first on purpose — the message must still put + // high-priority product_code ahead of low-priority infra. + asks: [ + { evidenceType: "infra", priority: "low", ask: { what: "pod status" } }, + { evidenceType: "product_code", priority: "high", ask: { what: "diff" } }, + ], + }; + } + return resolved("chat:2"); + }; + + await runRcaLoop({ + testRunId: "2", + firstMessage: "start", + submit, + config: CONFIG, + manifest: MANIFEST, + gather, + }); + + assert.equal( + submits[1].message, + "BLOCK:product_code\n\nBLOCK:infra", + "high-priority product_code must precede low-priority infra regardless of which gather resolved first", + ); +}); + +test("gap blocks still follow every gathered block, unaffected by concurrency", async () => { + const gather = async (g) => `BLOCK:${g.evidenceType}`; + + let calls = 0; + const submits = []; + const submit = async (args) => { + calls++; + submits.push(args); + if (calls === 1) { + return { + status: "NEEDS_INFO", + threadId: "chat:3", + asks: [ + { evidenceType: "product_code", priority: "high", ask: { what: "diff" } }, + { evidenceType: "metrics", priority: "low", ask: { what: "latency" } }, // no capability -> gap + ], + }; + } + return resolved("chat:3"); + }; + + const result = await runRcaLoop({ + testRunId: "3", + firstMessage: "start", + submit, + config: CONFIG, + manifest: MANIFEST, + gather, + }); + + assert.match(submits[1].message, /^BLOCK:product_code\n\nASK:/); + assert.deepEqual(result.asks_fulfilled, ["product_code"]); + assert.deepEqual(result.asks_unavailable, ["metrics"]); +}); diff --git a/tests/loop-turn1-result.test.mjs b/tests/loop-turn1-result.test.mjs new file mode 100644 index 0000000..a7d286f --- /dev/null +++ b/tests/loop-turn1-result.test.mjs @@ -0,0 +1,110 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { runRcaLoop } from "../lib/loop.mjs"; + +// Step 4b (SKILL.md Step 4b) pre-submits turn 1 for a cluster representative, +// concurrently with Step 4's evidence pre-fetch. When it lands NEEDS_INFO, +// `turn1Result` carries that thread + those asks in so the loop never +// resubmits turn 1 — this file is the conformance coverage for that skip-ahead +// path, mirroring tests/conformance.test.mjs's fixture style but inline since +// there's nothing to replay: turn 1 already happened before runRcaLoop starts. + +const CONFIG = { + turnCap: 6, + evidenceRouting: { + test_logs: { owner: "tfa", skip: true }, + product_code: { capability: "github" }, + other: { capability: "other" }, + }, +}; +const GITHUB_AVAILABLE = { github: { available: true, via: "gh" } }; +const gather = async (g) => `ASK: ${g.ask.what}\nTYPE: ${g.evidenceType}\nFOUND: yes\nSUMMARY: stub`; + +test("turn1Result: never resubmits turn 1, starts at ROUTE, turns_used counts the pre-dispatched turn", async () => { + const submits = []; + // The only submit() call the loop makes is the FOLLOW-UP — turn 1 already + // happened in Step 4b and is represented purely by `turn1Result`. + const submit = async (args) => { + submits.push(args); + return { + status: "RESOLVED", + threadId: "chat:99", + confidence: "high", + glimpse: { root_cause: "root cause found", failure_type: "product_regression", related_prs: ["#1"] }, + viewRca: "https://automation.browserstack.com/x", + }; + }; + + const result = await runRcaLoop({ + testRunId: "99", + submit, + config: CONFIG, + manifest: GITHUB_AVAILABLE, + gather, + turn1Result: { threadId: "chat:99", asks: [{ evidenceType: "product_code", ask: { what: "diff" } }] }, + }); + + assert.equal(submits.length, 1, "turn 1 must never be submitted — only the follow-up"); + assert.equal(submits[0].threadId, "chat:99", "the follow-up reuses Step 4b's thread"); + assert.equal(submits[0].turnId, undefined, "no turnId — NEEDS_INFO never carries one"); + assert.equal(result.status, "RESOLVED"); + assert.equal(result.turns_used, 2, "1 = Step 4b's pre-dispatched turn, 2 = this follow-up"); + assert.equal(result.threadId, "chat:99"); +}); + +test("turn1Result only short-circuits the FIRST pass — later iterations submit normally", async () => { + let calls = 0; + const submit = async () => { + calls++; + if (calls === 1) { + return { status: "NEEDS_INFO", threadId: "chat:99", asks: [{ evidenceType: "other", ask: { what: "logs excerpt" } }] }; + } + return { + status: "RESOLVED", + threadId: "chat:99", + confidence: "medium", + glimpse: { root_cause: "resolved on turn 3", failure_type: "infra", related_prs: [] }, + viewRca: "https://automation.browserstack.com/y", + }; + }; + + const result = await runRcaLoop({ + testRunId: "99", + submit, + config: CONFIG, + manifest: GITHUB_AVAILABLE, + gather, + turn1Result: { threadId: "chat:99", asks: [{ evidenceType: "product_code", ask: { what: "diff" } }] }, + }); + + assert.equal(calls, 2, "one real submit for the ROUTE follow-up, one more to resolve"); + assert.equal(result.status, "RESOLVED"); + assert.equal(result.turns_used, 3, "1 pre-dispatched + 2 real submits"); +}); + +test("without turn1Result, behaviour is unchanged: turn 1 IS submitted normally", async () => { + const submits = []; + const submit = async (args) => { + submits.push(args); + return { + status: "RESOLVED", + threadId: "chat:1", + confidence: "high", + glimpse: { root_cause: "root", failure_type: "product_regression", related_prs: [] }, + viewRca: "https://automation.browserstack.com/z", + }; + }; + + const result = await runRcaLoop({ + testRunId: "1", + firstMessage: "Initiating collaborative RCA for test run 1.", + submit, + config: CONFIG, + manifest: GITHUB_AVAILABLE, + gather, + }); + + assert.equal(submits.length, 1); + assert.equal(submits[0].message, "Initiating collaborative RCA for test run 1."); + assert.equal(result.turns_used, 1); +}); diff --git a/tests/turn1-registry.test.mjs b/tests/turn1-registry.test.mjs new file mode 100644 index 0000000..ab053f9 --- /dev/null +++ b/tests/turn1-registry.test.mjs @@ -0,0 +1,128 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync, statSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + turn1PathFor, + initTurn1Registry, + recordTurn1, + readTurn1, + readAllTurn1, + deleteTurn1Registry, +} from "../lib/turn1-registry.mjs"; + +const mode = (p) => statSync(p).mode & 0o777; + +function fixture() { + return mkdtempSync(join(tmpdir(), "rca-t1-")); +} + +test("turn1PathFor keys the file by buildId under the given stateDir", () => { + const dir = fixture(); + const p = turn1PathFor("build-123", dir); + assert.equal(p, join(dir, "rca-turn1.build-123.json")); + rmSync(dir, { recursive: true, force: true }); +}); + +test("readTurn1 on a non-existent registry returns null, not a throw", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + assert.equal(readTurn1(p, "3900000001"), null); + rmSync(dir, { recursive: true, force: true }); +}); + +test("initTurn1Registry creates the file and is idempotent (never clobbers prior entries)", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + initTurn1Registry(p, "b1", 1000); + assert.ok(existsSync(p)); + + recordTurn1(p, "3900000001", { status: "NEEDS_INFO", threadId: "chat:1", asks: ["a"] }, 2000); + // Re-init after entries exist must leave them untouched. + initTurn1Registry(p, "b1", 3000); + assert.deepEqual(readTurn1(p, "3900000001").asks, ["a"]); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("recordTurn1 + readTurn1 round-trip a PENDING entry", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + recordTurn1(p, "3900000002", { status: "PENDING", threadId: "chat:2", turnId: "t-2" }, 5000); + + const entry = readTurn1(p, "3900000002"); + assert.equal(entry.status, "PENDING"); + assert.equal(entry.threadId, "chat:2"); + assert.equal(entry.turnId, "t-2"); + assert.equal(entry.submittedAtMs, 5000); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("recordTurn1 + readTurn1 round-trip a NEEDS_INFO entry", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + recordTurn1( + p, + "3900000003", + { status: "NEEDS_INFO", threadId: "chat:3", asks: [{ evidenceType: "product_code" }] }, + 6000, + ); + + const entry = readTurn1(p, "3900000003"); + assert.equal(entry.status, "NEEDS_INFO"); + assert.equal(entry.threadId, "chat:3"); + assert.deepEqual(entry.asks, [{ evidenceType: "product_code" }]); + assert.equal(entry.turnId, undefined, "NEEDS_INFO never carries a turnId"); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("readAllTurn1 returns every recorded entry keyed by testRunId", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + recordTurn1(p, "1", { status: "PENDING", threadId: "chat:1", turnId: "t-1" }, 1000); + recordTurn1(p, "2", { status: "NEEDS_INFO", threadId: "chat:2", asks: [] }, 2000); + + const all = readAllTurn1(p); + assert.deepEqual(Object.keys(all).sort(), ["1", "2"]); + assert.equal(all["1"].status, "PENDING"); + assert.equal(all["2"].status, "NEEDS_INFO"); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("recordTurn1 for a second testRunId does not clobber the first", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + recordTurn1(p, "1", { status: "PENDING", threadId: "chat:1", turnId: "t-1" }, 1000); + recordTurn1(p, "2", { status: "PENDING", threadId: "chat:2", turnId: "t-2" }, 2000); + + assert.equal(readTurn1(p, "1").threadId, "chat:1"); + assert.equal(readTurn1(p, "2").threadId, "chat:2"); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("the registry file and its directory are owner-only (0600 / 0700)", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + recordTurn1(p, "1", { status: "PENDING", threadId: "chat:1", turnId: "t-1" }, 1000); + + assert.equal(mode(p), 0o600); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("deleteTurn1Registry removes the file and reports whether it existed", () => { + const dir = fixture(); + const p = turn1PathFor("b1", dir); + assert.equal(deleteTurn1Registry(p), false, "nothing to delete yet"); + + recordTurn1(p, "1", { status: "PENDING", threadId: "chat:1", turnId: "t-1" }, 1000); + assert.equal(deleteTurn1Registry(p), true); + assert.equal(existsSync(p), false); + + rmSync(dir, { recursive: true, force: true }); +}); diff --git a/workflows/rca-batch.mjs b/workflows/rca-batch.mjs index 4992bee..9d04a92 100644 --- a/workflows/rca-batch.mjs +++ b/workflows/rca-batch.mjs @@ -32,7 +32,16 @@ export const meta = { // // dispatch prompt (as before) is exactly the duplication this file // // removes. // clusters: [ -// { cluster_id, representative: { testRunId, testName, error_summary }, +// { cluster_id, +// representative: { testRunId, testName, error_summary, +// // Step 4b pre-dispatch outcome (SKILL.md Step 4b, lib/turn1-registry.mjs) +// // — at most one of these two is set, never both: +// turn1: { status: "PENDING", threadId, turnId } | +// { status: "NEEDS_INFO", threadId, asks }, +// // Step 4b's turn 1 already RESOLVED — no dispatch at all for this +// // representative; `resolved` is the RCA_SCHEMA-shaped result to use +// // directly (also already flipped into the CSV by the orchestrator). +// resolved: <RCA_SCHEMA object> | undefined }, // siblings: [ { testRunId, testName, error_summary } ] } // ] // } @@ -89,14 +98,56 @@ function resumeLine(row) { ].filter(Boolean).join("\n"); } +// Step 4b (SKILL.md Step 4b) already submitted this representative's turn 1, +// concurrently with Step 4's evidence pre-fetch. RESOLVED needs no coordinator +// dispatch at all (short-circuited in the pipeline stage below); these two +// non-terminal outcomes are handed to the coordinator instead of letting it +// submit turn 1 again — mutually exclusive per agents/ai-tfa-coordinator.md. +function turn1Line(r) { + const t = r?.turn1; + if (!t) return null; + if (t.status === "PENDING" && t.turnId) { + return [ + `RESUME (turn 1 already submitted by Step 4b — do not start a new thread):`, + `threadId=${t.threadId} turnId=${t.turnId}.`, + `Call getTfaTurnResult(testRunId, turnId) FIRST to read its current state`, + `(drain any soft-PENDING per the softPendingDrain budget) before submitting`, + `anything further — reuse this threadId for every follow-up on this test.`, + ].join("\n"); + } + if (t.status === "NEEDS_INFO") { + return [ + `TURN 1 ALREADY SUBMITTED AND ANSWERED by Step 4b — do NOT submit turn 1 again.`, + `threadId=${t.threadId}. turns_used starts at 1.`, + `TFA's turn-1 response was NEEDS_INFO with these asks (verbatim): ${JSON.stringify(t.asks ?? [])}`, + `Start this run at the ROUTE-the-asks step using them, then submit your first`, + `follow-up message on this SAME thread.`, + ].join("\n"); + } + return null; +} + function repPrompt(cluster) { const r = cluster.representative; + const resume = resumeLine(r); + // Mutual exclusivity, enforced in code, not just by convention: a + // representative gets AT MOST one resume-style instruction. A prior-run CSV + // pending-resume (`r.threadId`/`r.turnId`, an already in-flight thread from + // a run this build is resuming) takes precedence over a same-run Step 4b + // entry (`r.turn1`) — Step 4b's pre-dispatch is supposed to skip a + // representative already in pending-resume (SKILL.md Step 4b), but this is + // the backstop: presenting BOTH would hand the coordinator two different + // threadIds as "the" thread to resume, which is worse than picking one. + const t1 = resume ? null : turn1Line(r); return [ `You are the ai-tfa-coordinator for cluster ${cluster.cluster_id}.`, - `Run the FULL collaborative RCA loop for the representative test.`, + t1 + ? `Turn 1 was pre-dispatched by Step 4b — see below for how to resume it. Otherwise run the FULL collaborative RCA loop for the representative test.` + : `Run the FULL collaborative RCA loop for the representative test.`, `testRunId=${r.testRunId} testName=${r.testName ?? ""}`, `error_digest: ${r.error_summary ?? "(none)"}`, - resumeLine(r), + resume, + t1, shared, `Return the structured RCA_OUTPUT for this test.`, ].filter(Boolean).join("\n"); @@ -131,12 +182,18 @@ log(`Batch: ${clusters.length} cluster(s) over build ${ctx.buildId ?? "?"}`); const results = await pipeline( clusters, (cluster) => - agent(repPrompt(cluster), { - label: `rep:${cluster.representative.testRunId}`, - phase: "Representatives", - agentType: "tfa-rca:ai-tfa-coordinator", - schema: RCA_SCHEMA, - }).then((rca) => ({ cluster, rca })), + // Step 4b's turn 1 already RESOLVED this representative — no dispatch at + // all, zero added latency. The orchestrator already flipped this row's + // CSV entry to terminal; `resolved` just needs to flow into the sibling + // stage's pre_seed the same way a dispatched rep's result would. + cluster.representative?.resolved + ? Promise.resolve({ cluster, rca: cluster.representative.resolved }) + : agent(repPrompt(cluster), { + label: `rep:${cluster.representative.testRunId}`, + phase: "Representatives", + agentType: "tfa-rca:ai-tfa-coordinator", + schema: RCA_SCHEMA, + }).then((rca) => ({ cluster, rca })), ({ cluster, rca }) => parallel( (cluster.siblings ?? []).map((sib) => () => From c7b2ded8cf71865469b736f6111070cb68489d0d Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 00:33:32 +0530 Subject: [PATCH 40/51] feat(rca): delete a build's temp/registry artifacts after a successful report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cleanupBuildArtifacts(buildId, stateDir) removes this build's own CSV, evidence file + .contrib shards, tool cache dir, and turn1 registry — called only after triggerRcaReport succeeds (Step 6), never a periodic sweep. Safe specifically because Step 6 only runs once every row is terminal: there is nothing left to resume for this build at that point. Deliberately separate from lib/state-dir.mjs's pruneStateDir, which remains the age-based safety net for a build that crashes before ever reaching Step 6. --- lib/build-cleanup.mjs | 59 ++++++++++++++++++++++++ tests/build-cleanup.test.mjs | 88 ++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 lib/build-cleanup.mjs create mode 100644 tests/build-cleanup.test.mjs diff --git a/lib/build-cleanup.mjs b/lib/build-cleanup.mjs new file mode 100644 index 0000000..e8faab7 --- /dev/null +++ b/lib/build-cleanup.mjs @@ -0,0 +1,59 @@ +// Deletes ONE build's own temp/registry artifacts once its RCA report has +// generated successfully (skills/rca-build/SKILL.md Step 6, after +// triggerRcaReport succeeds — never before, never on a partial/failed run). +// +// This is deliberately NOT lib/state-dir.mjs's pruneStateDir: that function is +// a periodic, age-based sweep across every build in the shared temp dir, kept +// manual because it can't tell a finished build from an abandoned one and +// deleting a resumable build's state would silently break `pending-resume`. +// This module never faces that ambiguity — it only runs for a build whose +// every CSV row is already terminal and whose report just generated, so there +// is nothing left in this build's own state to resume. `pruneStateDir` still +// exists as the safety net for builds that never reach Step 6 (crashed +// mid-run); wiring that in is a separate, unrelated concern. +// +// Deletes exactly the four artifact families a build can produce, all scoped +// by buildId so a concurrent run over a DIFFERENT build in the same stateDir +// is never touched: +// - rca-state.<buildId>.csv (lib/csv-state.mjs) +// - rca-evidence.<buildId>.json (+ .contrib/) (lib/evidence-file.mjs) +// - rca-toolcache.<buildId>/ (lib/tool-cache.mjs) +// - rca-turn1.<buildId>.json (lib/turn1-registry.mjs) + +import { existsSync, rmSync } from "node:fs"; +import { csvPathFor } from "./csv-state.mjs"; +import { evidencePathFor, contribDirFor } from "./evidence-file.mjs"; +import { toolCacheDirFor } from "./tool-cache.mjs"; +import { turn1PathFor } from "./turn1-registry.mjs"; + +/** + * Delete this build's own CSV, evidence file + contribution shards, tool + * cache, and turn1 registry. Best-effort per path: a missing file is not an + * error (not every build produces a turn1 registry, e.g.), and a delete that + * throws (permissions, vanished mid-sweep) is recorded in `errors` rather than + * aborting the rest of the cleanup. + * + * Returns `{ deleted: [paths], errors: [{path, message}] }`. + */ +export function cleanupBuildArtifacts(buildId, stateDir = "") { + const targets = [ + csvPathFor(buildId, stateDir), + evidencePathFor(buildId, stateDir), + contribDirFor(evidencePathFor(buildId, stateDir)), + toolCacheDirFor(buildId, stateDir), + turn1PathFor(buildId, stateDir), + ]; + + const deleted = []; + const errors = []; + for (const path of targets) { + if (!existsSync(path)) continue; + try { + rmSync(path, { recursive: true, force: true }); + deleted.push(path); + } catch (err) { + errors.push({ path, message: err?.message ?? String(err) }); + } + } + return { deleted, errors }; +} diff --git a/tests/build-cleanup.test.mjs b/tests/build-cleanup.test.mjs new file mode 100644 index 0000000..e903052 --- /dev/null +++ b/tests/build-cleanup.test.mjs @@ -0,0 +1,88 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync, existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { cleanupBuildArtifacts } from "../lib/build-cleanup.mjs"; +import { csvPathFor } from "../lib/csv-state.mjs"; +import { evidencePathFor, contribDirFor } from "../lib/evidence-file.mjs"; +import { toolCacheDirFor } from "../lib/tool-cache.mjs"; +import { turn1PathFor } from "../lib/turn1-registry.mjs"; + +function fixture() { + return mkdtempSync(join(tmpdir(), "rca-cleanup-")); +} + +// Writes every artifact family for one build, so a test can assert cleanup +// removed exactly (and only) what that build produced. +function seedBuild(buildId, dir) { + writeFileSync(csvPathFor(buildId, dir), "buildId,testRunId\n"); + writeFileSync(evidencePathFor(buildId, dir), "{}"); + const contrib = contribDirFor(evidencePathFor(buildId, dir)); + mkdirSync(contrib, { recursive: true }); + writeFileSync(join(contrib, "3900000001.json"), "{}"); + const cache = toolCacheDirFor(buildId, dir); + mkdirSync(cache, { recursive: true }); + writeFileSync(join(cache, "entry.json"), "{}"); + writeFileSync(turn1PathFor(buildId, dir), "{}"); +} + +test("cleanupBuildArtifacts deletes every artifact family for the given build", () => { + const dir = fixture(); + seedBuild("b1", dir); + + const r = cleanupBuildArtifacts("b1", dir); + + assert.equal(existsSync(csvPathFor("b1", dir)), false); + assert.equal(existsSync(evidencePathFor("b1", dir)), false); + assert.equal(existsSync(contribDirFor(evidencePathFor("b1", dir))), false); + assert.equal(existsSync(toolCacheDirFor("b1", dir)), false); + assert.equal(existsSync(turn1PathFor("b1", dir)), false); + assert.equal(r.deleted.length, 5); + assert.deepEqual(r.errors, []); + + rmSync(dir, { recursive: true, force: true }); +}); + +// The load-bearing test: a concurrent run over a DIFFERENT build in the same +// stateDir must survive this build's cleanup untouched — nothing here should +// ever glob/sweep the shared directory. +test("cleanupBuildArtifacts never touches a different build's artifacts in the same stateDir", () => { + const dir = fixture(); + seedBuild("b1", dir); + seedBuild("b2", dir); + + cleanupBuildArtifacts("b1", dir); + + assert.equal(existsSync(csvPathFor("b1", dir)), false); + assert.equal(existsSync(csvPathFor("b2", dir)), true, "other build's CSV must survive"); + assert.equal(existsSync(evidencePathFor("b2", dir)), true, "other build's evidence file must survive"); + assert.equal(existsSync(contribDirFor(evidencePathFor("b2", dir))), true, "other build's shards must survive"); + assert.equal(existsSync(toolCacheDirFor("b2", dir)), true, "other build's tool cache must survive"); + assert.equal(existsSync(turn1PathFor("b2", dir)), true, "other build's turn1 registry must survive"); + + rmSync(dir, { recursive: true, force: true }); +}); + +test("cleanupBuildArtifacts is a no-op, not a throw, when nothing was ever written for this build", () => { + const dir = fixture(); + const r = cleanupBuildArtifacts("never-ran", dir); + assert.deepEqual(r, { deleted: [], errors: [] }); + rmSync(dir, { recursive: true, force: true }); +}); + +test("cleanupBuildArtifacts tolerates a build with only some artifact families present", () => { + const dir = fixture(); + // Only the CSV and turn1 registry exist — no evidence file, no tool cache + // (e.g. an unclustered rerun that never hit Step 4b's NEEDS_INFO/PENDING path). + writeFileSync(csvPathFor("b1", dir), "buildId,testRunId\n"); + writeFileSync(turn1PathFor("b1", dir), "{}"); + + const r = cleanupBuildArtifacts("b1", dir); + + assert.equal(r.deleted.length, 2); + assert.equal(existsSync(csvPathFor("b1", dir)), false); + assert.equal(existsSync(turn1PathFor("b1", dir)), false); + + rmSync(dir, { recursive: true, force: true }); +}); From 4fd5a5485e9864f72468b83af4c47d0aa6141330 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 00:38:22 +0530 Subject: [PATCH 41/51] feat(rca): server-side failure-theme clustering (Step 3), preferred over client-side signatures clustersFromThemes(rows, themesResult, testsByThemeId) builds { rows, clusters } from getBuildFailureThemes + listTestsInFailureTheme (server-computed themes), mirroring lib/signature.mjs's clusterRows() shape so downstream fan-out code doesn't care which path produced it. Falls back to lib/signature.mjs's client-side text-signature clustering when getBuildFailureThemes reports ready: false. A row already claimed by an earlier theme is skipped (first-theme-wins) rather than landing in two clusters; any failed test the server didn't assign to a theme becomes its own singleton cluster, never silently dropped. --- lib/theme-clustering.mjs | 71 +++++++++ skills/rca-build/references/clustering.md | 32 ++++- tests/theme-clustering.test.mjs | 168 ++++++++++++++++++++++ 3 files changed, 267 insertions(+), 4 deletions(-) create mode 100644 lib/theme-clustering.mjs create mode 100644 tests/theme-clustering.test.mjs diff --git a/lib/theme-clustering.mjs b/lib/theme-clustering.mjs new file mode 100644 index 0000000..3826ef8 --- /dev/null +++ b/lib/theme-clustering.mjs @@ -0,0 +1,71 @@ +// Server-computed failure-theme clustering (o11y `buildThemes`/`flat` via the +// getBuildFailureThemes/listTestsInFailureTheme MCP tools). Preferred over +// lib/signature.mjs's client-side text-signature clustering whenever the +// server-side computation is ready (skills/rca-build/SKILL.md Step 3). Falls +// back to lib/signature.mjs when getBuildFailureThemes reports `ready: false`. +// +// Pure + dependency-free, mirroring lib/signature.mjs's shape and +// testability: takes already-fetched plain data in, returns { rows, clusters } +// with the same shape clusterRows() produces, so downstream code (the fan-out +// workflow, the sequential harness) doesn't care which path produced it. + +import { selectRepresentative } from "./signature.mjs"; + +// Build { rows, clusters } from a getBuildFailureThemes result (`ready: true`) +// plus a per-theme map of already-fetched member rows (keyed by +// buildFailureThemeId, each entry the array listTestsInFailureTheme returned +// for that theme, already paginated to completion). `rows` is the full +// listTestIds row set — used to enrich each theme member with the row's own +// testName/error_summary and to catch any failed test the server didn't +// assign to a theme: never silently dropped, it becomes its own singleton, +// same convention as lib/signature.mjs. Mutates each row's `cluster_id`, same +// as `clusterRows()`. +// +// Themes are expected to be disjoint (a test belongs to at most one), but +// this isn't a guarantee the server's contract documents — so a row already +// claimed by an earlier theme is skipped (first-theme-wins) rather than +// letting it land in two clusters with conflicting `cluster_id` values. +export function clustersFromThemes(rows, themesResult, testsByThemeId) { + const rowById = new Map(rows.map((r) => [String(r.testRunId), r])); + const covered = new Set(); + const clusters = []; + + for (const theme of themesResult.buildThemes ?? []) { + const themeRows = (testsByThemeId[theme.buildFailureThemeId] ?? []) + .map((t) => rowById.get(String(t.testRunId))) + .filter((r) => r && !covered.has(String(r.testRunId))); + + if (themeRows.length === 0) continue; + + const id = `theme-${theme.buildFailureThemeId}`; + themeRows.forEach((r) => { + r.cluster_id = id; + covered.add(String(r.testRunId)); + }); + + const representative = selectRepresentative(themeRows); + const siblings = themeRows.filter((m) => m !== representative); + clusters.push({ + cluster_id: id, + signature: theme.themeData?.name ?? "", + members: themeRows, + representative, + siblings, + }); + } + + for (const row of rows) { + if (covered.has(String(row.testRunId))) continue; + const id = `solo-${row.testRunId}`; + row.cluster_id = id; + clusters.push({ + cluster_id: id, + signature: "", + members: [row], + representative: row, + siblings: [], + }); + } + + return { rows, clusters }; +} diff --git a/skills/rca-build/references/clustering.md b/skills/rca-build/references/clustering.md index 1ac5866..689434d 100644 --- a/skills/rca-build/references/clustering.md +++ b/skills/rca-build/references/clustering.md @@ -1,4 +1,4 @@ -# Failure-signature clustering +# Clustering Why: a red build's N failures usually trace to a handful of causes (one bad PR/deploy/shared helper). Running the full collaborative loop once per *cause* @@ -7,9 +7,33 @@ causes)** — the only thing that makes "RCA for ALL failed tests, even thousand feasible. But **every failed test must still show a per-test RCA in the TRA dashboard**, so clustering collapses the *evidence hunt*, not the *output*. -The logic lives in `lib/signature.mjs`; this file is the protocol. - -## The signature +## Two sources, one contract + +Both paths produce the identical `{ cluster_id, signature, members, +representative, siblings }` shape, so nothing downstream (the fan-out +workflow, the sequential harness) needs to know which one ran: + +- **Preferred — server-computed themes.** `lib/theme-clustering.mjs` → + `clustersFromThemes(rows, themesResult, testsByThemeId)`, fed from the + `getBuildFailureThemes` / `listTestsInFailureTheme` MCP tools (SKILL.md Step + 3). `getBuildFailureThemes` is responsible for making themes exist, not just + reading them — if nothing has ever been computed for this build it triggers + computation (one POST, same call) and polls for `buildThemeWorkflow.status` + to reach `SUCCESS`, up to its own budget. The grouping reflects the + server's own root-cause clustering instead of a text-signature guess — two + failures with an identical error string but unrelated causes are not + conflated the way a client-side "signature" would be. +- **Fallback — client-side failure signature.** `lib/signature.mjs` → + `clusterAndPersist(csvPath, csvStateModule)`, the original text-normalization + approach described below. Used whenever `getBuildFailureThemes` comes back + `ready: false` — still computing past its poll budget, a failure status, or + `status: "trigger-unavailable"` (the trigger call didn't succeed). This + makes the flow independent of whether the trigger call currently succeeds: + whenever it doesn't, every un-computed build degrades straight to this + fallback; whenever it does, the same call reaches `ready: true` on its own + and this fallback simply isn't exercised. + +## The signature (fallback path only) Computed from the trimmed failure detail `listTestIds(includeFailureDetail=true)` already returns on each row — **no extra probe turns**: diff --git a/tests/theme-clustering.test.mjs b/tests/theme-clustering.test.mjs new file mode 100644 index 0000000..eb1004b --- /dev/null +++ b/tests/theme-clustering.test.mjs @@ -0,0 +1,168 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { clustersFromThemes } from "../lib/theme-clustering.mjs"; + +function row(id, extra = {}) { + return { + testRunId: String(id), + failure_category: "Assertion", + error_summary: "expected 200 but got 500", + file_path: "spec/login.rb", + is_flaky: "false", + ...extra, + }; +} + +function theme(buildFailureThemeId, name = "Some Theme") { + return { + themeId: `uuid-${buildFailureThemeId}`, + buildFailureThemeId, + themeData: { name, description: "..." }, + affectedWorkflows: [], + }; +} + +test("one theme with two members → one cluster, representative + one sibling", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [theme(10, "Data Assertion Mismatch")] }; + const testsByThemeId = { 10: [{ testRunId: "1" }, { testRunId: "2" }] }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 1); + assert.equal(clusters[0].cluster_id, "theme-10"); + assert.equal(clusters[0].signature, "Data Assertion Mismatch"); + assert.equal(clusters[0].members.length, 2); + assert.equal(clusters[0].siblings.length, 1); + assert.ok(clusters[0].representative); +}); + +test("multiple themes → one cluster each", () => { + const rows = [row(1), row(2), row(3)]; + const themesResult = { + buildThemes: [theme(10, "Theme A"), theme(20, "Theme B")], + }; + const testsByThemeId = { + 10: [{ testRunId: "1" }], + 20: [{ testRunId: "2" }, { testRunId: "3" }], + }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 2); + const a = clusters.find((c) => c.cluster_id === "theme-10"); + const b = clusters.find((c) => c.cluster_id === "theme-20"); + assert.equal(a.members.length, 1); + assert.equal(a.siblings.length, 0); + assert.equal(b.members.length, 2); + assert.equal(b.siblings.length, 1); +}); + +test("a failed test not assigned to any theme becomes its own singleton (never dropped)", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [theme(10, "Theme A")] }; + const testsByThemeId = { 10: [{ testRunId: "1" }] }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 2); + const solo = clusters.find((c) => c.cluster_id === "solo-2"); + assert.ok(solo, "uncovered test must still get a cluster"); + assert.equal(solo.members.length, 1); + assert.equal(solo.siblings.length, 0); + assert.equal(solo.representative.testRunId, "2"); +}); + +test("a theme with no matched member rows is skipped, not an empty cluster", () => { + const rows = [row(1)]; + const themesResult = { buildThemes: [theme(10, "Theme A"), theme(20, "Empty theme")] }; + const testsByThemeId = { 10: [{ testRunId: "1" }], 20: [] }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 1); + assert.equal(clusters[0].cluster_id, "theme-10"); +}); + +test("member rows not present in listTestIds rows are dropped, not fabricated", () => { + const rows = [row(1)]; + const themesResult = { buildThemes: [theme(10, "Theme A")] }; + // testsByThemeId names a testRunId ("999") that never appeared in listTestIds. + const testsByThemeId = { 10: [{ testRunId: "1" }, { testRunId: "999" }] }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 1); + assert.equal(clusters[0].members.length, 1); +}); + +test("representative selection matches lib/signature.mjs's rule (non-flaky, then smallest testRunId)", () => { + const rows = [ + row(5, { is_flaky: "true" }), + row(9, { is_flaky: "false" }), + row(7, { is_flaky: "false" }), + ]; + const themesResult = { buildThemes: [theme(10)] }; + const testsByThemeId = { + 10: [{ testRunId: "5" }, { testRunId: "9" }, { testRunId: "7" }], + }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters[0].representative.testRunId, "7"); +}); + +test("clustersFromThemes stamps cluster_id onto every row, theme and singleton alike", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [theme(10)] }; + const testsByThemeId = { 10: [{ testRunId: "1" }] }; + + clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(rows[0].cluster_id, "theme-10"); + assert.equal(rows[1].cluster_id, "solo-2"); +}); + +test("numeric testRunId in theme membership (as the MCP tool's JSON would send it) still matches string testRunId rows", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [theme(10)] }; + // Membership entries carry testRunId as a NUMBER, unlike listTestIds rows (strings). + const testsByThemeId = { 10: [{ testRunId: 1 }, { testRunId: 2 }] }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 1); + assert.equal(clusters[0].members.length, 2); +}); + +test("a testRunId claimed by an earlier theme is skipped by a later theme (first-theme-wins, no duplicate membership)", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [theme(10, "Theme A"), theme(20, "Theme B")] }; + // Both themes claim testRunId "1" — the server is expected never to do this, + // but the function must not let the row land in two clusters. + const testsByThemeId = { + 10: [{ testRunId: "1" }], + 20: [{ testRunId: "1" }, { testRunId: "2" }], + }; + + const { clusters } = clustersFromThemes(rows, themesResult, testsByThemeId); + + assert.equal(clusters.length, 2); + const a = clusters.find((c) => c.cluster_id === "theme-10"); + const b = clusters.find((c) => c.cluster_id === "theme-20"); + assert.equal(a.members.length, 1); + assert.equal(a.members[0].testRunId, "1"); + assert.equal(b.members.length, 1, "testRunId 1 must not also land in theme 20"); + assert.equal(b.members[0].testRunId, "2"); + assert.equal(rows[0].cluster_id, "theme-10"); +}); + +test("no themes at all → every row is its own singleton", () => { + const rows = [row(1), row(2)]; + const themesResult = { buildThemes: [] }; + + const { clusters } = clustersFromThemes(rows, themesResult, {}); + + assert.equal(clusters.length, 2); + assert.ok(clusters.every((c) => c.cluster_id.startsWith("solo-"))); +}); From ad2cc7ccb9757c13d0229c5021501352ee39a143 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 00:39:51 +0530 Subject: [PATCH 42/51] docs(rca): document server-side theme clustering, Step 4b pre-dispatch, and Step 5/6 wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two bodies of work landed together in this file (interleaved at the hunk level, so split by content below rather than by commit): - Step 3: document the server-side clustersFromThemes path (getBuildFailureThemes + listTestsInFailureTheme) as preferred over lib/signature.mjs's client-side clustering, with the fallback condition and verification step. - Step 4b (new): turn-1 pre-dispatch for cluster representatives, concurrent with Step 4's evidence pre-fetch — mechanic, the three-way branch on RESOLVED/NEEDS_INFO/PENDING, and the skip condition for a representative already in pending-resume state. - Step 5: wires the turn1 registry into each representative's dispatch (resume for PENDING, turn1_result for NEEDS_INFO); rewrites the "per cluster, not global" sibling-ordering guidance into an explicit rolling work-queue, distinguishing the Workflow path's true per-cluster streaming from the direct-Agent-tool-dispatch path's per-batch limit; adds an explicit warning that the direct-dispatch path has no code enforcing the Step 4b handoff, unlike the other two execution paths. - Step 6: calls cleanupBuildArtifacts only after triggerRcaReport succeeds. - Hard rules: carves out Step 4b's direct tfaRcaTurn call as the sole, narrow exception to "always dispatch via ai-tfa-coordinator". - API reference: documents lib/turn1-registry.mjs and lib/build-cleanup.mjs's exports (enforced by tests/wiring.test.mjs's doc-drift guard). --- skills/rca-build/SKILL.md | 371 ++++++++++++++++++++++++++++++-------- 1 file changed, 299 insertions(+), 72 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 65a54b1..e2ab31b 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -11,15 +11,17 @@ logs; the client agent owns everything else** (product code, infra/runtime, logs metrics, deploy, ci) — routed by capability, generic over product and infra. This skill is the **build-level orchestrator** (`ai-tfa-orchestrator` role). It -never calls `tfaRcaTurn` itself — it dispatches the `ai-tfa-coordinator` -(test-level) per test/cluster member, which drives the loop and lets TFA author -the dashboard RCA. **The full RCA report lives on the Test Observability UI, not -in Claude** — this run's job is to feed it, then surface a terse glimpse and the +dispatches the `ai-tfa-coordinator` (test-level) per test/cluster member, which +drives the loop and lets TFA author the dashboard RCA — the one narrow +exception is Step 4b's turn-1 pre-dispatch, a single direct `tfaRcaTurn` call +per cluster representative, concurrent with Step 4. **The full RCA report +lives on the Test Observability UI, not in Claude** — this run's job is to feed it, then surface a terse glimpse and the link. There is exactly **one mode**: autonomous. There is exactly **one gate** (Step -1) before execution. After the gate closes, **the run never asks the user -anything again.** + +1. before execution. After the gate closes, **the run never asks the user + anything again.** Config (concurrency, turn-cap, paths, evidence registry) lives in `config/rca.config.json`. State lives in the CSV/WAL spine (`lib/csv-state.mjs`). @@ -78,6 +80,23 @@ hardenStateDir(dir) run once at gate start; idempotent pruneStateDir(dir, nowMs, {maxAgeMs, dryRun}) NOT automatic — these files are the resume state ``` +**Step 4b turn-1 pre-dispatch registry — `lib/turn1-registry.mjs`** +``` +turn1PathFor(buildId, stateDir="") → <stateDir|tmpdir>/bstack-rca/rca-turn1.<buildId>.json +initTurn1Registry(path, buildId, nowMs) idempotent, never clobbers existing entries +recordTurn1(path, testRunId, {status, threadId, turnId?, asks?}, nowMs) PENDING or NEEDS_INFO only — RESOLVED is flipped straight into the CSV instead +readTurn1(path, testRunId) → entry | null +readAllTurn1(path) → {testRunId: entry} run-end stats only +deleteTurn1Registry(path) → boolean (existed?) called by lib/build-cleanup.mjs +``` + +**Build-completion cleanup — `lib/build-cleanup.mjs`** +``` +cleanupBuildArtifacts(buildId, stateDir="") → {deleted, errors} + deletes THIS build's CSV, evidence file + .contrib shards, tool cache dir, and turn1 registry. + Call ONLY after triggerRcaReport succeeds (Step 6) — never a periodic sweep, see lib/state-dir.mjs. +``` + **Routing / output — `lib/routing.mjs`, `lib/glimpse.mjs`, `lib/evidence-cache.mjs`** ``` loadConfig(configPath) buildManifest(config, discovered) routeAsks(asks, config, manifest) @@ -150,7 +169,7 @@ that grep the wrong repos on the wrong branch. **Disambiguating by product (nudge / one-question rule).** Connector skills are product-scoped — a workspace may hold none, one, or several product families (e.g. `<product-a>-*`, `<product-b>-*`, whatever the user has). After the `ls`, -pick the *product family* whose connector skills apply to THIS build: +pick the _product family_ whose connector skills apply to THIS build: - **Zero families found** → **nudge the user in the gate summary**: "No connector-shaped skills found under `.claude/skills/` — proceeding with @@ -168,12 +187,12 @@ pick the *product family* whose connector skills apply to THIS build: has already run, else defer this to a re-visit after discovery) against each family's declared file paths / error patterns — if one family owns the failure surface, use it. - If both signals leave the choice ambiguous, this earns the **one - consolidated gate question** (Part B rules apply): fold it into the same - question as any other non-assumable field, e.g. *"Multiple product families - found (`<product-a>`, `<product-b>`); build/failure signatures don't - uniquely pick one — which family owns this build's failures?"* Headless: - pick the first alphabetically and record the ambiguity as a gap. + If both signals leave the choice ambiguous, this earns the **one + consolidated gate question** (Part B rules apply): fold it into the same + question as any other non-assumable field, e.g. _"Multiple product families + found (`<product-a>`, `<product-b>`); build/failure signatures don't + uniquely pick one — which family owns this build's failures?"_ Headless: + pick the first alphabetically and record the ambiguity as a gap. Then enumerate every connector relevant to test RCA: @@ -186,17 +205,17 @@ Then enumerate every connector relevant to test RCA: **Validate** each with a cheap probe — discovery alone is not enough: -| Connector | Probe | -|---|---| -| github | `gh auth status` (or a GitHub MCP tool listed) | -| infra | ANY runtime connector the user has — probe what exists, never assume one: `kubectl version --request-timeout=5s`, `docker ps`, `aws ecs list-clusters`, `nomad status`, `pm2 ls`, or an infra-shaped skill/MCP tool. Record the KIND in the manifest (`via: kubectl \| docker \| ecs \| …`) | -| logs | a log-search skill/MCP tool actually listed in the session | -| metrics | a metrics skill/MCP tool actually listed in the session | -| other | best-effort; default `absent` | +| Connector | Probe | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| github | `gh auth status` (or a GitHub MCP tool listed) | +| infra | ANY runtime connector the user has — probe what exists, never assume one: `kubectl version --request-timeout=5s`, `docker ps`, `aws ecs list-clusters`, `nomad status`, `pm2 ls`, or an infra-shaped skill/MCP tool. Record the KIND in the manifest (`via: kubectl \| docker \| ecs \| …`) | +| logs | a log-search skill/MCP tool actually listed in the session | +| metrics | a metrics skill/MCP tool actually listed in the session | +| other | best-effort; default `absent` | **Scope validation — run the SCOPE PROBES declared by each connector skill.** The base probe above (`gh auth status`, `kubectl version`, …) only confirms the -raw tool works. It does not confirm the *concrete targets* a coordinator will +raw tool works. It does not confirm the _concrete targets_ a coordinator will touch — specific repos, branches, clusters, namespaces, indices — are actually reachable. That is the connector SKILL's job: each connector skill MUST declare, in its `Capability declaration` section, a `Scope probes:` list @@ -238,7 +257,7 @@ is the point: - cheap inference (e.g. the automation repo is the cwd if it holds the tests). **Product-repo corroboration (do NOT skip).** The product repo must plausibly -be the *system under test for THIS build's failures* — not merely a repo name +be the _system under test for THIS build's failures_ — not merely a repo name found lying around. A repo mentioned only in workspace docs/READMEs is a **weak hint, never an assumption**: cross-check it against the failure signatures (discovery runs first if needed) — do the failing area, files, or error strings @@ -248,6 +267,7 @@ the doc-sourced repo is discarded — never carry it (or its PRs) into the manifest as a settled product repo. When corroboration leaves **no** product repo, decide by whether a human can help: + - **PRs were supplied** → treat those as the suspect surface; product repo is derived from them. No question needed. - **No PRs, interactive session** → the product repo is now **non-assumable AND @@ -265,9 +285,9 @@ non-assumable AND load-bearing. In practice that set is: the build id; **the product repo when it could not be corroborated and no PRs were supplied** (see above — without it the culprit-PR hunt cannot run); and rarely an ambiguous repo when PRs were supplied. Those, and only those, may be asked **ONCE, in a single -consolidated question at gate close** — e.g. *"Failures look like `<domain>`; +consolidated question at gate close** — e.g. _"Failures look like `<domain>`; which repo owns that code? (reply 'none' → I'll RCA without culprit-PR -attribution)."* Never a second question. **Headless: skip asking entirely; +attribution)."_ Never a second question. **Headless: skip asking entirely; record the gaps.** ### Gate close @@ -317,41 +337,71 @@ Re-running `seed` on an existing CSV is idempotent and preserves terminal rows (resume-safe — same build id → same path). If `listTestIds` returns empty → write an empty CSV, report "no failed tests", stop. -## Step 3 — failure-signature clustering (see references/clustering.md) - -Use **`clusterAndPersist(csvPath, csvStateModule)`** (`lib/signature.mjs`), not -`clusterRows` directly: - -```js -const clusters = clusterAndPersist(csvPath, await import("./lib/csv-state.mjs")); -``` - -`clusterRows` assigns `cluster_id` **in place** and returns `{rows, clusters}`, -so `const { clusters } = clusterRows(rows)` gives you working cluster objects -while every `cluster_id` is silently discarded — the CSV keeps empty cluster -columns and the run degrades to **one coordinator per test**, losing the whole -representative/sibling collapse. That is measured, not theoretical: a real run -went 12 tests → 26 subagents and 30 minutes with the clustering "done" but -never written. `clusterAndPersist` writes back and verifies the count, so it -cannot forget. - -Then verify before fan-out: **if `cluster_id` is empty on any row, Step 3 did -not take effect** — do not proceed, the run would silently cost O(tests) -instead of O(causes). +## Step 3 — clustering (see `<pluginRoot>/skills/rca-build/references/clustering.md`) -Compute a failure signature per row and assign `cluster_id` (`lib/signature.mjs`). Each cluster gets one **representative** (full multi-turn loop) and `N−1` **siblings** (pre-seeded one-turn confirm against their own logs). This collapses the expensive evidence hunt to O(distinct causes) while every test still lands a per-test RCA. Singleton clusters are just plain per-test loops. -## Step 4 — build-evidence pre-fetch (see references/evidence-routing.md and lib/evidence-file.mjs) +**Prefer the server's own clustering over recomputing it client-side:** + +1. Call `getBuildFailureThemes(buildUuid=<build id>)`. If nothing has ever + been computed for this build, this triggers computation (one POST, same + call) and polls in-call up to its own budget for `buildThemeWorkflow.status` + to reach `SUCCESS`. +2. **`ready: true`** → for each entry in `buildThemes`, call + `listTestsInFailureTheme(buildUuid=<build id>, themeId=<buildFailureThemeId>)`, + following `nextCursor` until exhausted, to get that theme's member + testRunIds. Feed the `listTestIds` rows + the themes result + the per-theme + member lists into `lib/theme-clustering.mjs` → `clustersFromThemes(rows, +themesResult, testsByThemeId)` — this is the **preferred path**, since the + grouping reflects the server's own root-cause analysis rather than a text- + signature guess, and it never runs the coordinator fan-out N-tests-wide for + a build with only a handful of distinct causes duplicated across teams. + Any failed test the server didn't assign to a theme is never dropped — it + still gets its own singleton cluster. +3. **`ready: false`** (still computing past the poll budget, a failure status, + or `status: "trigger-unavailable"` — the trigger call didn't succeed) → + **fall back** to **`clusterAndPersist(csvPath, csvStateModule)`** + (`lib/signature.mjs`), not `clusterRows` directly: + + ```js + const clusters = clusterAndPersist(csvPath, await import("./lib/csv-state.mjs")); + ``` + + `clusterRows` assigns `cluster_id` **in place** and returns `{rows, clusters}`, + so `const { clusters } = clusterRows(rows)` gives you working cluster objects + while every `cluster_id` is silently discarded — the CSV keeps empty cluster + columns and the run degrades to **one coordinator per test**, losing the whole + representative/sibling collapse. That is measured, not theoretical: a real run + went 12 tests → 26 subagents and 30 minutes with the clustering "done" but + never written. `clusterAndPersist` writes back and verifies the count, so it + cannot forget. + + Never block the run waiting on the server; the fallback keeps the same + `{ cluster_id, representative, siblings }` contract the fan-out consumes, + so nothing downstream needs to know which path produced it. This also + makes the flow independent of whether the trigger call currently succeeds: + whenever it doesn't, `getBuildFailureThemes` degrades to `ready: false` + fast (no wasted poll budget) and this fallback engages every time; + whenever it does, the same call reaches `ready: true` on its own and this + fallback simply isn't exercised — no code change required either way. + +`clustersFromThemes` mutates each row's `cluster_id` in place but does NOT +persist — it's pure/dependency-free by design. Write its rows back yourself +with `csvState.writeRows(csvPath, rows)` before fan-out; `clusterAndPersist` +already does this for the fallback path. Then verify either way: **if +`cluster_id` is empty on any row, Step 3 did not take effect** — do not +proceed, the run would silently cost O(tests) instead of O(causes). + +## Step 4 — build-evidence pre-fetch (see `<pluginRoot>/skills/rca-build/references/evidence-routing.md` and `<pluginRoot>/lib/evidence-file.mjs`) Once, after clustering (Step 3) and before fan-out — the capability manifest already exists from Gate Part A, reuse it, do not re-discover. This step replaces each coordinator's own turn-1 evidence sweep with ONE pre-fetch: -it does not remove the requirement that turn-1 evidence exists, only *who -gathers it*. +it does not remove the requirement that turn-1 evidence exists, only _who +gathers it_. 1. Resolve the evidence-file path: `lib/evidence-file.mjs` → `evidencePathFor(buildId, config.paths.stateDir)` — @@ -366,14 +416,23 @@ gathers it*. routing table (never one workload guessed from the first failing test). 3. For each repo: run the connector skill's PR-window-search + deploy-state recipes **once**, using `lib/evidence-cache.mjs`'s `compute(repo, range, - evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. +evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. Digest the result into the `evidence-block.md` shape, then persist via `setGithubEvidence(path, repo, {deployState, prsInWindow, gap}, nowMs)`. A repo the connector can't reach records `{gap: "<reason>"}` — never blocks the rest of the pre-fetch. - **Ask for `files` in the PR-list call — it costs nothing extra and is the - single highest-leverage thing in this step:** + **`--json` on THIS FIRST `gh pr list` call MUST include `files` — there is + no separate step where it gets added later.** This is the single + highest-leverage thing in Step 4, and it is a MUST, not a nice-to-have: a + PR-list call than omits `files` here is never corrected downstream — it + just becomes one `gh pr view <n> --json files` per PR, run from inside the + `for pr in ...` loop this exact mistake produces. Measured on a real run: + the orchestrator listed PRs without `files` (5 `gh pr list` calls), then + looped `gh pr view --json files` once per PR to backfill it (9 calls) — + 100% avoidable had the first call carried `files`. There is no legitimate + reason to split these into two calls; `--json files` costs nothing extra + on the list call itself. ```bash gh pr list -R <org>/<repo> --state merged --base <branch> \ @@ -386,11 +445,16 @@ gathers it*. fetches were **44 of 407 gh calls (10.8%)** — all of them avoidable here. Store the paths in each PR's `files` field rather than leaving it `null`: path-overlap is the first falsification test in - `references/github-evidence.md`, so with `files` populated a coordinator + `<pluginRoot>/skills/rca-build/references/github-evidence.md`, so with `files` populated a coordinator rules a suspect in or out from the evidence file alone, and only fetches a diff for the handful that survive. Do NOT pre-fetch diffs — those are large and only a few PRs ever need one. + A per-PR `gh pr view --json files` call is legitimate ONLY for a suspect + PR discovered later (during a coordinator's own investigation, not in this + pre-fetch's window) — never as a backfill for a PR-list call that should + have carried `files` the first time. + Two other measured wastes this step should pre-empt: - **Never let coordinators re-probe connectors.** `gh auth status` / `kubectl version` accounted for **17 of 407 gh calls (4.2%)** purely @@ -403,7 +467,7 @@ gathers it*. 4. For each workload: run the connector skill's compulsory kubectl + VictoriaLogs sweep **once**, anchored to the build's own clock — never "now". **PAD the window: `started_at − 2m` .. `finished_at + 10m`.** - `finished_at` is when the build was *marked* finished, which is not when + `finished_at` is when the build was _marked_ finished, which is not when the failing behaviour stopped: on one real build an upstream outage began at 06:12:00 and ran to 06:15:51, while `finished_at` was 06:12:21 — a sweep scoped strictly to `started_at..finished_at` saw 21 seconds of a @@ -412,11 +476,11 @@ gathers it*. coordinator can weigh it; do NOT silently widen to an arbitrary window (that is the separate, opposite failure of matching a coincidence from unrelated traffic). Persist via `setLogsEvidence(path, workload, - {clusterIds, kubectlSweep, victorialogs, gap}, nowMs)`. +{clusterIds, kubectlSweep, victorialogs, gap}, nowMs)`. Two query mechanics that cost real calls when missed: - **`direction` defaults to newest-first**, so a limited query always - returns the END of the window. To find when something *started* — the + returns the END of the window. To find when something _started_ — the first request after a gap, the onset of an error burst — pass `direction: "forward"`. A gap "confirmed" from a backward query is not confirmed at all; it is just the tail of the range. @@ -424,6 +488,7 @@ gathers it*. from a wrong selector. Before reporting "no traffic", prove the logger was alive in the same window with a query you expect to be non-empty (e.g. readiness probes from a named pod). Only then is silence evidence. + 5. `resolveBaseline(lastGreenRef, fallbackRef)` (from `lib/evidence-cache.mjs`) → `setBaseline(path, baseline, suspectWindow, nowMs)`. No "last green" baseline (never-green suite) → fall back to a configured baseline ref and @@ -477,6 +542,77 @@ by path-overlap relevance, not every PR in the window. Pass `evidencePathFor(...)`'s path to Step 5's fan-out as `evidenceFilePath` — every dispatch (representative and sibling) must be told to read it first. +## Step 4b — turn-1 pre-dispatch (runs CONCURRENTLY with Step 4, not after it) + +Every cluster's representative testRunId is already known the moment Step 3 +finishes — Step 4b does not wait for Step 4's evidence pre-fetch, because +turn 1's message has no dependency on it: it is built entirely from Step 2's +CSV seed (`error_summary`/`testName`), exactly the same construction +`agents/ai-tfa-coordinator.md`'s loop step 0 uses when neither `pre_seed` nor +`resume` applies (`error_digest` present → `"Error: <title + endpoint>"`; else +→ `"Initiating collaborative RCA for test run <id>."`). Nothing here needs the +evidence file, so there is no ordering hazard in running the two concurrently. + +**Mechanic:** in the SAME tool-call batch as Step 4's evidence-gathering calls +(`gh`, `kubectl`, MCP queries), ALSO call `tfaRcaTurn(testRunId=<rep>, +message=<first-turn digest>)` directly — one call per cluster representative — +as additional calls in that batch, so they execute concurrently with Step 4's +own work rather than sequentially before or after it. This is a direct MCP +call from the orchestrator, not a coordinator dispatch: turn 1 alone is cheap +enough that spinning up a full `ai-tfa-coordinator` subagent for it would cost +more than the latency it saves. + +1. `initTurn1Registry(turn1PathFor(buildId, config.paths.stateDir), buildId, nowMs)` + once, before submitting any turn 1s (`lib/turn1-registry.mjs`). +2. **Skip any representative whose CSV row already has a `threadId` + + `turnId`** (a `pending-resume` row from a prior run attempt — an already + in-flight thread). Submitting a fresh turn 1 for it would start a SECOND + thread for the same test, which every other part of this contract + (`agents/ai-tfa-coordinator.md`'s "one thread per test" hard limit) forbids. + That representative resumes its existing thread at Step 5 exactly as + before Step 4b existed — Step 4b only ever applies to a representative with + no prior thread at all. +3. For every remaining (thread-less) cluster representative, submit turn 1 and + branch on the result: + - **RESOLVED** → `flip()` this CSV row straight to terminal, right here — + same fields a coordinator's `RCA_OUTPUT` would set (`rca_done: resolved`, + `root_cause`, `failure_type`, `related_prs`, `view_rca`, `confidence`, + `turns_used: 1`, `threadId`). This representative needs **no Step 5 + dispatch at all** — the cheapest possible outcome. **Do not wait for + Step 5 to formally start: dispatch this cluster's siblings immediately, + right here in Step 4b**, via `siblingPreSeed(csvPath, csvState, + clusterId, representativeId)` against the row you just flipped — a + sibling only ever needs its OWN representative's result, never the state + of any other cluster, so nothing about Step 5's fan-out has to begin + first. This is the ONLY case a sibling can be dispatched this early, and + the reason is narrow: it works because the representative resolved in + ONE pre-dispatched turn, so `pre_seed` is already real evidence, not a + guess. A representative still mid-loop (`NEEDS_INFO`/`PENDING`) has no + `root_cause` yet — dispatching that cluster's siblings before it lands + would degrade every one of them into a full independent investigation + (Step 5's measured cost note: 22.7 tool calls/2.2 turns vs 8.0/2.0 for a + representative, one run 60 calls/17 minutes). Never do that; siblings of + a not-yet-resolved representative wait for Step 5 exactly as documented + there. + - **NEEDS_INFO** → `recordTurn1(path, testRunId, {status: "NEEDS_INFO", + threadId, asks}, nowMs)`. A real, non-terminal answer — hand it to Step + 5's coordinator as `turn1_result` (never resubmit turn 1). + - **PENDING** → `recordTurn1(path, testRunId, {status: "PENDING", threadId, + turnId}, nowMs)`. Do **not** drain it here — Step 4 is running + concurrently and there is no reason to block Step 4b on it. Step 5's + coordinator dispatch already knows how to drain a soft-PENDING (the + existing `resume` input covers this case as-is). +4. Nothing about this starts a second thread: it is exactly turn 1 of the one + thread the Step 5 coordinator continues from `threadId`. + +Pass `turn1PathFor(...)`'s path to Step 5 alongside `evidenceFilePath` — Step 5 +must read it (`readTurn1(path, testRunId)`) before building each +representative's dispatch and translate the result into the matching input: +`PENDING` → `resume: {threadId, turnId}`; `NEEDS_INFO` → `turn1_result: +{threadId, asks}`; a flipped-to-terminal row (no registry entry, CSV already +`resolved`) → no dispatch, use the CSV row's result directly as this cluster's +representative outcome for seeding siblings. + ## Step 5 — fan-out (fully autonomous) **ORDER MATTERS: representative first, siblings only after it lands.** For each @@ -504,6 +640,35 @@ Drive the cluster work-list, **`concurrency` (default 20) at a time**: representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL (claim → heartbeat → flip) so the run is resumable. +**"Per cluster, not global" is a rolling work-queue, not two rigid phases.** +Do NOT dispatch "all representatives first, then all siblings" as two fixed +mega-batches — that reintroduces a global-ish wait: any cluster's siblings +would sit idle until every representative in the current batch lands, not just +their own. Instead, whenever a batch of dispatches returns, immediately refill +the next batch by mixing (a) siblings of whichever representatives just +resolved (via `siblingPreSeed`) with (b) any not-yet-dispatched representatives +from other clusters, up to `concurrency` slots — so a fast cluster's siblings +enter the very next batch instead of waiting out an unrelated slow +representative. + +This distinction matters differently on each path: +- **Opt-in `workflows/rca-batch.mjs`** achieves this structurally, for free: + `pipeline(clusters, repStage, siblingStage)` has NO barrier between stages — + a cluster's siblings start the instant ITS OWN representative resolves, + fully interleaved with every other cluster's progress. Nothing to get wrong + here. +- **Default direct Agent-tool dispatch** cannot be sub-batch-streaming the same + way, because a single assistant turn's parallel tool calls are a real + synchronization point: the orchestrator does not regain control until every + call in that turn's batch has returned. So within any one batch, a cluster + whose representative resolves early still cannot dispatch its siblings until + the WHOLE batch drains — the rolling-refill discipline above is what keeps + that batch-local wait from becoming a build-wide one, but it cannot eliminate + it entirely. **When cluster count exceeds `concurrency`, or when the + Workflow tool is available, prefer `workflows/rca-batch.mjs`** for + latency-sensitive builds — it is the only path with a true per-cluster (not + per-batch) guarantee. + > **Concurrency comes from `config/rca.config.json` — always read it from > there, never hardcode.** The default path (direct Agent-tool dispatch) honors > the JSON value literally: fan out coordinator subagents in batches of @@ -516,12 +681,29 @@ representatives deep, siblings one-turn-confirm. Eagerly persist to the CSV/WAL - **Default (all hosts, including Claude Code) → direct Agent-tool dispatch.** Read `concurrency` from `config/rca.config.json` and dispatch `tfa-rca:ai-tfa-coordinator` subagents in batches of that size (one message, - up to `concurrency` tool-use blocks per batch). This path is **outside the - Workflow runtime**, so the `min(16, cores-2)` ceiling does not apply and the - JSON value is honored literally. Prefer this path whenever the machine's - Workflow cap (`min(16, cores-2)`) would be smaller than the configured - `concurrency` — e.g. an 8-core Mac caps Workflow at 6 while the JSON asks - for 20. + up to `concurrency` tool-use blocks per batch), refilling each next batch per + the rolling work-queue discipline above — never two rigid all-reps / + all-siblings phases. This path is **outside the Workflow runtime**, so the + `min(16, cores-2)` ceiling does not apply and the JSON value is honored + literally. Prefer this path whenever the machine's Workflow cap (`min(16, + cores-2)`) would be smaller than the configured `concurrency` — e.g. an + 8-core Mac caps Workflow at 6 while the JSON asks for 20 — but remember it + only gets per-BATCH streaming, not per-cluster: prefer + `workflows/rca-batch.mjs` instead whenever cluster count exceeds + `concurrency` and the Workflow tool is available. + + **This path has no code enforcing the Step 4b handoff — you are the + enforcement.** Unlike `workflows/rca-batch.mjs` (which reads the registry in + code via `turn1Line()`) and `lib/loop.mjs` (which takes `turn1Result` as a + structural parameter), building a representative's dispatch prompt here is + entirely on you. **Before dispatching ANY representative, call + `readTurn1(turn1PathFor(buildId, stateDir), testRunId)` and fold the result + into the prompt** (`resume`/`turn1_result` per `agents/ai-tfa-coordinator.md`) + — and skip the dispatch entirely if the CSV row is already `resolved`. Omit + this and Step 4b's pre-dispatch is silently wasted: the coordinator submits + turn 1 again on a brand-new thread, abandoning the one Step 4b already + started (not incorrect — the run still resolves — just the entire latency + win thrown away without any error to notice it by). - Opt-in `workflows/rca-batch.mjs` (Claude Code only) → use only when the Workflow tool's structured `pipeline()`/`parallel()` orchestration, `resumeFromRunId` resumability, or progress UI is worth the concurrency @@ -536,11 +718,33 @@ coordinator that dies becomes a recorded `failed` row — one stuck test never sinks the batch (partial-first). No path ever prompts the user (the gate is closed). +**Coordinator prompts MUST carry `pluginRoot` and use it to fully qualify every +reference-doc / lib path.** A coordinator is dispatched fresh, with no +guarantee about its own cwd — `references/evidence-routing.md` (bare, +relative) resolves against whatever directory the coordinator happens to +start in, which is routinely NOT this plugin's root. Measured: 12 calls +across coordinators were `Read` attempts at the wrong bare path followed by a +`find` to recover the real one (`<pluginRoot>/skills/rca-build/references/evidence-routing.md`, +`.../github-evidence.md`, `.../clustering.md`). Every dispatch prompt must +state `pluginRoot=<absolute path>` up front and every reference-doc pointer in +the prompt (and echoed from `agents/ai-tfa-coordinator.md`) must already be +`pluginRoot`-qualified — never a bare `references/<file>.md`. + +**Coordinator prompts MUST also point at the API reference instead of letting +the coordinator re-derive it.** State plainly in the dispatch prompt: "Function +signatures for `lib/*.mjs` are documented at `<pluginRoot>/skills/rca-build/SKILL.md` +§ API reference — read that section once if a signature is needed; do not +`grep`/`Read`/`cat` the `lib/` source to re-derive a signature already +documented there." Measured: 15 of 49 self-discovery-tax calls were a +coordinator re-deriving a `lib/*.mjs` signature from source (one coordinator +read `lib/evidence-file.mjs` three times plus one `grep`, all to re-learn +`contributeLogsEvidence`'s signature) — a cost this pointer removes. + **Coordinator prompts MUST name every connector-shaped skill on the manifest.** Each dispatch prompt lists, per capability, the resolved connector skill from -Gate Part A Step 0 — e.g. *"Use `<resolved-github-skill>` for every +Gate Part A Step 0 — e.g. _"Use `<resolved-github-skill>` for every product_code / deploy / ci ask (canonical repos + branch live in the skill; do -NOT grep other repos). Use `<resolved-infra-skill>` for every infra ask."* A +NOT grep other repos). Use `<resolved-infra-skill>` for every infra ask."_ A coordinator prompt that omits a manifest-listed connector skill — and that therefore lets the coordinator infer repos from workspace `git remote` or cwd — is a bug: the @@ -548,15 +752,15 @@ coordinator will land plausible-but-wrong PR attributions on adjacent repos. **Coordinator prompts MUST also name the Step 4 evidence file.** Every dispatch prompt (representative and sibling alike) includes the absolute -`evidenceFilePath` from Step 4 with the instruction: *"Read `<path>` (via the +`evidenceFilePath` from Step 4 with the instruction: _"Read `<path>` (via the Read tool) before making any live github/infra/logs gather call. It's a pre-fetch, not a hard dependency — a repo/workload it doesn't name, or marks with a `gap`, is a genuine gap: fall back to the capability manifest above -exactly as if no file existed."* For a sibling, add: *"The file's data about +exactly as if no file existed."_ For a sibling, add: _"The file's data about your OWN test's workload is real evidence, not inheritance — reading it is fine. What must stay independent is the CONFIRMATION judgment: never adopt the representative's verdict just because the file already has the answer in -it."* A dispatch prompt that omits this path forces its coordinator back into +it."_ A dispatch prompt that omits this path forces its coordinator back into a full independent sweep — exactly the redundancy Step 4 exists to remove. **The file is read-write, not just read-only.** When a coordinator has to @@ -588,8 +792,8 @@ complementary (the file is read wholesale at turn 1, the cache answers a specific repeat query later). **Also hand every dispatch the tool cache.** The evidence file shares digested -*findings*; `bin/cached-exec.mjs` / `bin/cached-mcp.mjs` share raw *call -results*, which is where most duplicate work actually hides — on one measured +_findings_; `bin/cached-exec.mjs` / `bin/cached-mcp.mjs` share raw _call +results_, which is where most duplicate work actually hides — on one measured build `gh` was 37% of all coordinator tool calls and 46 were byte-identical commands re-run by different coordinators. Include the plugin root in each dispatch prompt so coordinators can invoke the wrappers, and tell them to pass @@ -612,7 +816,7 @@ under this layout. **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the github connector (deploy timeline vs last-pass window, changed paths vs failure -signature — `references/github-evidence.md`) and feed the PR link(s) to TFA in +signature — `<pluginRoot>/skills/rca-build/references/github-evidence.md`) and feed the PR link(s) to TFA in the turn message so the dashboard RCA's `related_prs` populates. An application-bug RCA with no GitHub PR link is **incomplete**: keep digging until the turn cap; if still none, the turn must explicitly state "no culprit PR @@ -629,7 +833,18 @@ link — that is all. When every row is terminal: (`<N> tests · <R> resolved · <P> pending · <F> failed`). **Nothing per-test.** 2. Call **`triggerRcaReport(buildUuid=<build id>)`** (add `force=true` only to re-run over an existing completed report). -3. Print the link line, verbatim shape: +3. **Only once that call succeeds**, call + `cleanupBuildArtifacts(buildId, config.paths.stateDir)` + (`lib/build-cleanup.mjs`) to delete THIS build's own CSV, evidence file + + `.contrib/` shards, tool cache, and turn1 registry. Never call this before + `triggerRcaReport` succeeds, and never on a run that ends with any row still + non-terminal — at that point resume still needs these files. This is safe + specifically because Step 6 only runs "when every row is terminal": there is + nothing left to resume for THIS build once its report has generated. It is + deliberately not `lib/state-dir.mjs`'s `pruneStateDir` (a separate, manual, + age-based sweep across every build in the shared temp dir) — that remains + the safety net for a build that crashes before ever reaching Step 6. +4. Print the link line, verbatim shape: ``` Full report on the Test Observability UI: <viewReport> @@ -662,7 +877,12 @@ thread. the gate closes, never ask the user anything.** - An invalid/absent connector is a recorded gap, never a blocker. - Headless + missing build id → end immediately. Headless never asks. -- Never call `tfaRcaTurn` from this skill — always via the `ai-tfa-coordinator`. +- Never call `tfaRcaTurn` from this skill — always via the `ai-tfa-coordinator` — + **except Step 4b's turn-1 pre-dispatch**, which is a deliberate, narrow carve-out + (one direct call per cluster representative, concurrent with Step 4, never a + follow-up turn) documented there. Every OTHER `tfaRcaTurn` call — every turn + past 1, and every sibling's turn 1 — still goes exclusively through a + dispatched coordinator. - A soft-`PENDING` is never an answer: it must be drained with `getTfaTurnResult(testRunId, turnId)` before any further submit on that thread. Only a spent drain budget may end a test `PENDING`. @@ -672,3 +892,10 @@ thread. the Test Observability UI link only. - A PRODUCT_BUG RCA without a GitHub PR link is incomplete — dig until the turn cap, else state what was searched and record the gap. +- Step 4's first `gh pr list` call per repo MUST include `files` in `--json` — + never split into a plain list followed by a per-PR `gh pr view --json files` + backfill loop. +- Every reference-doc / `lib/` path handed to a coordinator (in the dispatch + prompt or in `agents/ai-tfa-coordinator.md`) MUST be `pluginRoot`-qualified + (`<pluginRoot>/skills/rca-build/references/<file>.md`) — never a bare + `references/<file>.md`, which resolves against an unknown coordinator cwd. From 9283ae1e8272e8a42f597514f1d44b2635dafff4 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 02:26:48 +0530 Subject: [PATCH 43/51] refactor: rewrite Step 4b as fire-and-forget async dispatch, parallelize Step 1 connector probes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 4b was previously documented as running "concurrently with Step 4" via a same-batch trick, but a real run executed it sequentially anyway because the wording didn't specify which of Step 4's several batches to merge into. Rewrite it around the Agent tool's genuine fire-and-forget semantics instead: dispatch one lightweight subagent per cluster representative for turn 1, proceed to Step 4 in the very next turn without waiting, and handle each result as pure bookkeeping as notifications interleave with Step 4's own turns. Adds: - TURN1_OUTPUT fixed-shape contract so concurrent dispatches stay attributable - concurrency-capped dispatch (reusing config's existing `concurrency` value) - fail-open documentation for a subagent that never reports back - immediate sibling pre-dispatch when a representative resolves in one turn - explicit "narrate as one combined phase" guidance to stop the same bug recurring via mis-narration Also applies the same independent/parallel-batching fix to Step 1 Gate Part A's connector base probes and scope probes, which had the same ambiguous sequential-looking wording. Pressure-tested via superpowers:writing-skills (fresh subagents given only the wording, no other context): Step 1 and two of three Step 4b scenarios executed correctly end to end. One gap found — an agent could still insert an unnecessary Step-4b-only turn before Step 4's own tool calls began, since nothing said Step 4b's prep (registry init, resume skip-list) has zero dependency on Step 4. Closed with an explicit sentence permitting/encouraging both in the same first turn. --- skills/rca-build/SKILL.md | 185 ++++++++++++++++++++++++++++++-------- 1 file changed, 146 insertions(+), 39 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index e2ab31b..039a262 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -203,7 +203,18 @@ Then enumerate every connector relevant to test RCA: - plus any connector-shaped skills / MCP servers present in the session (a log-search MCP, a metrics MCP, an infra skill, …). -**Validate** each with a cheap probe — discovery alone is not enough: +**Validate** each with a cheap probe — discovery alone is not enough. **Every +row below is independent of every other row — fire them all as one batch of +parallel tool calls, never one connector at a time.** A probe failing (or +being absent) never blocks another connector's probe from running; there is +nothing here for one row to wait on. For example: `gh auth status`, `kubectl +version --request-timeout=5s` (or whatever infra tool applies), a logs-MCP +check, and a metrics-MCP check all belong in the SAME turn — not four separate +turns run one after the other, and not "check github, then check infra, +then …". (This is the same class of bug Step 4b hit on a real run: a +sequential-*looking* list of independent checks got executed sequentially in +practice, costing minutes it never needed to. Don't repeat that here, at the +very front of the pipeline where it delays everything downstream.) | Connector | Probe | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -223,7 +234,16 @@ naming what to check and how. This orchestrator's contract is generic: 1. For every connector skill added to the manifest in Step 0, read its `Scope probes:` list. -2. Run each probe verbatim. +2. **Run every declared probe, across every connector and every target it + names, together in one batch — the same rule as the base probes above.** + The only real ordering constraint is *within* a single connector: its scope + probes are only worth running once that same connector's base probe has + passed (no point checking which repos github can reach if `gh auth status` + already failed). That is a per-connector dependency, not a global one — + e.g. github's repo-scope probes and infra's namespace-scope probes never + depend on each other, so they still fire in the same batch as soon as + their respective base probes clear. Never run one connector's scope + probes, wait for them, then move to the next connector's. 3. Record every target's result in the manifest entry — passes go into a resolved-scope field (e.g. `repos_validated: [...]`, `namespace: ok`), failures go into a per-target gap (e.g. `<target>: 404 not_accessible`). @@ -403,6 +423,15 @@ replaces each coordinator's own turn-1 evidence sweep with ONE pre-fetch: it does not remove the requirement that turn-1 evidence exists, only _who gathers it_. +**Narrate this as one combined phase, not two sequential ones.** Step 4b +(below) starts the moment Step 3 finishes and runs the whole time Step 4 does +— any progress line shown to the user during this window should say something +like `Evidence pre-fetch (Step 4) + turn-1 pre-dispatch (Step 4b)`, never "Step +4 done, now starting Step 4b." That sequential phrasing is exactly what caused +Step 4b to be *executed* sequentially in practice on a real run — the +narration and the execution went wrong together, and fixing only one of them +leaves the other free to reintroduce the bug. + 1. Resolve the evidence-file path: `lib/evidence-file.mjs` → `evidencePathFor(buildId, config.paths.stateDir)` — `<tmpdir>/bstack-rca/rca-evidence.<buildId>.json`, alongside the state CSV. @@ -542,68 +571,146 @@ by path-overlap relevance, not every PR in the window. Pass `evidencePathFor(...)`'s path to Step 5's fan-out as `evidenceFilePath` — every dispatch (representative and sibling) must be told to read it first. -## Step 4b — turn-1 pre-dispatch (runs CONCURRENTLY with Step 4, not after it) +## Step 4b — turn-1 pre-dispatch (fire-and-forget, fully async alongside Step 4) Every cluster's representative testRunId is already known the moment Step 3 -finishes — Step 4b does not wait for Step 4's evidence pre-fetch, because -turn 1's message has no dependency on it: it is built entirely from Step 2's -CSV seed (`error_summary`/`testName`), exactly the same construction +finishes, for however many clusters this build produced — never assume a +fixed count, it is whatever Step 3 found. Turn 1's message has no dependency +on Step 4's evidence pre-fetch at all: it is built entirely from Step 2's CSV +seed (`error_summary`/`testName`), exactly the same construction `agents/ai-tfa-coordinator.md`'s loop step 0 uses when neither `pre_seed` nor `resume` applies (`error_digest` present → `"Error: <title + endpoint>"`; else -→ `"Initiating collaborative RCA for test run <id>."`). Nothing here needs the -evidence file, so there is no ordering hazard in running the two concurrently. - -**Mechanic:** in the SAME tool-call batch as Step 4's evidence-gathering calls -(`gh`, `kubectl`, MCP queries), ALSO call `tfaRcaTurn(testRunId=<rep>, -message=<first-turn digest>)` directly — one call per cluster representative — -as additional calls in that batch, so they execute concurrently with Step 4's -own work rather than sequentially before or after it. This is a direct MCP -call from the orchestrator, not a coordinator dispatch: turn 1 alone is cheap -enough that spinning up a full `ai-tfa-coordinator` subagent for it would cost -more than the latency it saves. +→ `"Initiating collaborative RCA for test run <id>."`). So there is no need to +wait for Step 4 before starting Step 4b — and, just as importantly, no need to +wait for Step 4b either before moving on. + +**Mechanic: dispatch, don't wait.** For every cluster representative, launch +one lightweight subagent via the Agent tool whose ONLY job is to call +`tfaRcaTurn(testRunId=<rep>, message=<first-turn digest>)` once and emit one +fixed-shape block as its final output — no evidence gathering, no loop, no +drain. This is deliberately **not** a full `ai-tfa-coordinator` dispatch (that +agent's whole design is the multi-turn evidence-gathering loop, far more +machinery than "submit one message and return"); write a minimal, +purpose-built inline prompt for this instead, and put the exact output +contract below directly in that prompt — an Agent-tool result is free text, +and with many of these dispatched concurrently the orchestrator has no other +reliable way to tell which representative a given notification is even for. + +``` +TURN1_OUTPUT_START +testRunId: <the testRunId this subagent was given> +status: RESOLVED | NEEDS_INFO | PENDING +threadId: <threadId from the tfaRcaTurn response, or "none"> +turnId: <turnId — PENDING only, tfaRcaTurn never returns one for the other two statuses; else "none"> +glimpse: <RESOLVED only — the trimmed {root_cause, failure_type, related_prs, confidence, viewRca} object, verbatim; else "none"> +asks: <NEEDS_INFO only — the asks array, verbatim; else "none"> +TURN1_OUTPUT_END +``` + +That block — not prose, not a summary — is this subagent's entire final +message. It is exactly what the orchestrator reads back off the +task-notification to do the bookkeeping below: `status` selects the branch, +`testRunId` is the join key back to the right CSV row / registry entry, and +`threadId`/`turnId`/`glimpse`/`asks` are pasted straight into `flip()` or +`recordTurn1()` with no re-interpretation needed. + +An Agent-tool dispatch returns *immediately* with a launch confirmation, not +the subagent's result — this is fundamentally different from a batch of raw +MCP tool calls in one turn, which blocks the orchestrator until every call in +that turn returns. Fire off every representative's dispatch together, then +**immediately proceed to Step 4's evidence pre-fetch in the very next turn — +do not wait for any of them.** There is no "same batch as Step 4" trick to get +right here (an earlier version of this section relied on that and it is easy +to execute wrong, e.g. by finishing Step 4 first and only then starting Step +4b — the fire-and-forget dispatch here has no such ordering hazard, because +nothing about it requires being co-located with Step 4's own tool calls). + +As each subagent finishes — on its own schedule, bounded only by +`tfaRcaTurn`'s own ~90s in-call poll cap, so realistically within the first +minute or two of the run — a task-notification carrying its `TURN1_OUTPUT` +block arrives, interleaved with whichever Step 4 turn happens to be in flight +at that moment. Handle each one the moment you are next free to, as pure +bookkeeping — no new tool calls needed for this part: 1. `initTurn1Registry(turn1PathFor(buildId, config.paths.stateDir), buildId, nowMs)` - once, before submitting any turn 1s (`lib/turn1-registry.mjs`). + once, before dispatching any turn 1s (`lib/turn1-registry.mjs`). 2. **Skip any representative whose CSV row already has a `threadId` + `turnId`** (a `pending-resume` row from a prior run attempt — an already - in-flight thread). Submitting a fresh turn 1 for it would start a SECOND + in-flight thread). Dispatching a fresh turn 1 for it would start a SECOND thread for the same test, which every other part of this contract (`agents/ai-tfa-coordinator.md`'s "one thread per test" hard limit) forbids. That representative resumes its existing thread at Step 5 exactly as before Step 4b existed — Step 4b only ever applies to a representative with no prior thread at all. -3. For every remaining (thread-less) cluster representative, submit turn 1 and - branch on the result: +3. For every remaining (thread-less) cluster representative, dispatch its + turn-1 subagent. When its result notification lands, branch on it: - **RESOLVED** → `flip()` this CSV row straight to terminal, right here — same fields a coordinator's `RCA_OUTPUT` would set (`rca_done: resolved`, `root_cause`, `failure_type`, `related_prs`, `view_rca`, `confidence`, `turns_used: 1`, `threadId`). This representative needs **no Step 5 dispatch at all** — the cheapest possible outcome. **Do not wait for Step 5 to formally start: dispatch this cluster's siblings immediately, - right here in Step 4b**, via `siblingPreSeed(csvPath, csvState, - clusterId, representativeId)` against the row you just flipped — a - sibling only ever needs its OWN representative's result, never the state - of any other cluster, so nothing about Step 5's fan-out has to begin - first. This is the ONLY case a sibling can be dispatched this early, and - the reason is narrow: it works because the representative resolved in - ONE pre-dispatched turn, so `pre_seed` is already real evidence, not a - guess. A representative still mid-loop (`NEEDS_INFO`/`PENDING`) has no - `root_cause` yet — dispatching that cluster's siblings before it lands - would degrade every one of them into a full independent investigation - (Step 5's measured cost note: 22.7 tool calls/2.2 turns vs 8.0/2.0 for a - representative, one run 60 calls/17 minutes). Never do that; siblings of - a not-yet-resolved representative wait for Step 5 exactly as documented - there. + right here in Step 4b** — as their own fire-and-forget Agent-tool + dispatches too, same principle, don't wait on them either — via + `siblingPreSeed(csvPath, csvState, clusterId, representativeId)` against + the row you just flipped. A sibling only ever needs its OWN + representative's result, never the state of any other cluster, so + nothing about Step 5's fan-out has to begin first. This is the ONLY case + a sibling can be dispatched this early, and the reason is narrow: it + works because the representative resolved in ONE pre-dispatched turn, so + `pre_seed` is already real evidence, not a guess. A representative still + mid-loop (`NEEDS_INFO`/`PENDING`) has no `root_cause` yet — dispatching + that cluster's siblings before it lands would degrade every one of them + into a full independent investigation (Step 5's measured cost note: 22.7 + tool calls/2.2 turns vs 8.0/2.0 for a representative, one run 60 + calls/17 minutes). Never do that; siblings of a not-yet-resolved + representative wait for Step 5 exactly as documented there. - **NEEDS_INFO** → `recordTurn1(path, testRunId, {status: "NEEDS_INFO", threadId, asks}, nowMs)`. A real, non-terminal answer — hand it to Step 5's coordinator as `turn1_result` (never resubmit turn 1). - **PENDING** → `recordTurn1(path, testRunId, {status: "PENDING", threadId, - turnId}, nowMs)`. Do **not** drain it here — Step 4 is running - concurrently and there is no reason to block Step 4b on it. Step 5's - coordinator dispatch already knows how to drain a soft-PENDING (the - existing `resume` input covers this case as-is). + turnId}, nowMs)`. Do **not** drain it here — there is no reason to spend + any of the orchestrator's own time on it. Step 5's coordinator dispatch + already knows how to drain a soft-PENDING (the existing `resume` input + covers this case as-is). 4. Nothing about this starts a second thread: it is exactly turn 1 of the one thread the Step 5 coordinator continues from `threadId`. +5. **A subagent that never reports back fails open, not closed.** If a turn-1 + subagent dies, errors, or times out before emitting its `TURN1_OUTPUT` + block, no registry entry gets recorded for that representative — there is + nothing to distinguish "Step 4b never ran for this test" from "Step 4b ran + and failed." Both land in exactly the same place: Step 5's `readTurn1` + returns nothing, and Step 5 falls back to a completely normal, fresh + dispatch (submit turn 1 from scratch, no `resume`/`turn1_result`) — which + is functionally the retry. There is no separate "check Step 4b succeeded, + re-trigger turn 1 if not" step to build; the existing no-entry fallback + already covers it. The one real cost: if the dead subagent *did* reach + `tfaRcaTurn` before failing to report back, that thread is now orphaned — + Step 5's fresh dispatch starts a genuinely new thread rather than resuming + it. Not a correctness problem (the new thread resolves independently just + fine) — just one wasted, never-continued thread on TFA's side per failure. + +**This removes orchestrator-side blocking, not underlying capacity — cap the +fan-out itself.** Every dispatched subagent still makes a real `tfaRcaTurn` +call, consuming the same API/compute capacity Step 5's fan-out competes for. +"Async" means the orchestrator never sits idle waiting on these dispatches — +it does NOT mean the dispatches are free, and firing an unbounded number of +them at once for a build with many clusters risks the same session/rate-limit +cascade a large Step 5 fan-out can hit. **Dispatch at most `concurrency` (from +`config/rca.config.json` — the same value Step 5 already uses, not a separate +setting) turn-1 subagents at a time.** For a build with more cluster +representatives than that, issue the first `concurrency` immediately, then +issue the next batch as soon as they're dispatched (still fire-and-forget, +still never blocking Step 4's own progress) rather than firing every +representative in one shot regardless of cluster count. + +None of this — `initTurn1Registry`, the pending-resume skip-list check, or the +first dispatch batch — has any dependency on Step 4's own tool calls, or vice +versa. **The very first turn can contain Step 4b's setup-and-first-dispatch- +batch together with Step 4's own first evidence-gathering calls, in the same +batch.** Do not treat Step 4b's prep as a turn Step 4 waits behind, even for +one turn — that is the same one-extra-turn-of-latency mistake this whole +section exists to remove, just smaller. Pass `turn1PathFor(...)`'s path to Step 5 alongside `evidenceFilePath` — Step 5 must read it (`readTurn1(path, testRunId)`) before building each From aa32532ceddddbfc592f5c51cc079a49f6871503 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 02:35:46 +0530 Subject: [PATCH 44/51] =?UTF-8?q?fix(rca):=20restate=20Step=204b=E2=86=925?= =?UTF-8?q?=20resume/turn1=5Fresult=20mapping=20inline=20in=20Step=205?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pressure-tested every pre-today section of SKILL.md the same way as this session's Step 4b/Step 1 rewrite (superpowers:writing-skills, fresh subagents, verbatim section only). Step 3 (clustering path selection), Step 5 (rolling-refill work-queue, sibling guard), and Step 6 (cleanup-after-report gating) all executed correctly across every branch tested. One gap in Step 5: its own bullet only cross-referenced "resume/turn1_result per agents/ai-tfa-coordinator.md" without restating which registry status maps to which field. A fresh agent given only Step 5's text folded a NEEDS_INFO result into `resume: true` instead of the documented `turn1_result: {threadId, asks}` shape — the exact mapping only lived in Step 4b's own closing paragraph, which didn't survive being read in isolation. Restate the mapping explicitly in Step 5's own bullet so it doesn't depend on carrying detail forward from an earlier section. --- skills/rca-build/SKILL.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 039a262..fe3b102 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -805,12 +805,18 @@ This distinction matters differently on each path: structural parameter), building a representative's dispatch prompt here is entirely on you. **Before dispatching ANY representative, call `readTurn1(turn1PathFor(buildId, stateDir), testRunId)` and fold the result - into the prompt** (`resume`/`turn1_result` per `agents/ai-tfa-coordinator.md`) - — and skip the dispatch entirely if the CSV row is already `resolved`. Omit - this and Step 4b's pre-dispatch is silently wasted: the coordinator submits - turn 1 again on a brand-new thread, abandoning the one Step 4b already - started (not incorrect — the run still resolves — just the entire latency - win thrown away without any error to notice it by). + into the prompt using this exact mapping — the two are distinct coordinator + inputs (`agents/ai-tfa-coordinator.md`), never interchangeable:** + `PENDING` → `resume: {threadId, turnId}`; `NEEDS_INFO` → `turn1_result: + {threadId, asks}`; no registry entry with the CSV row already `resolved` → + skip the dispatch entirely, use the CSV row's result directly. Do NOT fold a + `NEEDS_INFO` result into a `resume` field, or vice versa — a coordinator + reads these as two different shapes and a swapped one is silently wrong, not + rejected. Omit this translation altogether and Step 4b's pre-dispatch is + silently wasted: the coordinator submits turn 1 again on a brand-new thread, + abandoning the one Step 4b already started (not incorrect — the run still + resolves — just the entire latency win thrown away without any error to + notice it by). - Opt-in `workflows/rca-batch.mjs` (Claude Code only) → use only when the Workflow tool's structured `pipeline()`/`parallel()` orchestration, `resumeFromRunId` resumability, or progress UI is worth the concurrency From ca8c64fc78245cab8f1c175e985d6a1eab486865 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 02:49:18 +0530 Subject: [PATCH 45/51] docs(rca): strip precise measured stats from SKILL.md, keep the qualitative rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every "measured: X of Y (Z%)" / "A vs B" citation was a one-time measurement from a specific past run, not a guarantee about the next one — the digits added token cost and false precision without changing the instruction. Replace each with a plain statement of the failure mode and, where it matters, a brief "this has happened on a real run" flag. The rule and its consequence are the load-bearing part; the exact arithmetic was decoration. --- skills/rca-build/SKILL.md | 129 +++++++++++++++++++------------------- 1 file changed, 64 insertions(+), 65 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index fe3b102..ca21852 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -28,11 +28,12 @@ Config (concurrency, turn-cap, paths, evidence registry) lives in ## API reference — read THIS, do not grep the source -Every signature this run needs, in one place. Measured reason it exists: on one -run **92 of 407 tool calls (23%, 5.1 per test)** were agents re-deriving this — -`grep -n "^export function" lib/…`, `cat config/…`, repeated `ls .claude/skills/`. -The previous run spent 13. The jump came from adding helpers faster than the -docs described them, so the plugin taxed every agent to learn itself. +Every signature this run needs, in one place. This exists because agents were +routinely re-deriving these signatures live — `grep -n "^export function" +lib/…`, `cat config/…`, repeated `ls .claude/skills/` — a real, recurring tax +that grew every time a helper was added faster than the docs described it, so +the plugin taxed every agent to relearn itself from source instead of reading +one page. Everything below is product-neutral: build ids, repos, branches, workloads and paths are all **inputs**, supplied by the gate and the connector skills. @@ -150,8 +151,8 @@ Run: # when this plugin is itself a repo inside the workspace, cwd is the plugin and # the product's connector skills sit one or two levels UP, so a bare # `ls .claude/skills/` finds nothing and the run silently degrades to raw MCP -# tools with best-effort repo guesses. Measured: it missed all three real -# connectors on this workspace. +# tools with best-effort repo guesses — on a real run it missed every +# connector actually present on the workspace this way. ls .claude/skills/ ../.claude/skills/ ../../.claude/skills/ ~/.claude/skills/ 2>/dev/null ``` @@ -394,10 +395,10 @@ themesResult, testsByThemeId)` — this is the **preferred path**, since the so `const { clusters } = clusterRows(rows)` gives you working cluster objects while every `cluster_id` is silently discarded — the CSV keeps empty cluster columns and the run degrades to **one coordinator per test**, losing the whole - representative/sibling collapse. That is measured, not theoretical: a real run - went 12 tests → 26 subagents and 30 minutes with the clustering "done" but - never written. `clusterAndPersist` writes back and verifies the count, so it - cannot forget. + representative/sibling collapse. This is a real failure mode, not a + theoretical one — a real run hit it, with the clustering silently "done" in + the return value but never written to the CSV. `clusterAndPersist` writes + back and verifies the count, so it cannot forget. Never block the run waiting on the server; the fallback keeps the same `{ cluster_id, representative, siblings }` contract the fan-out consumes, @@ -456,12 +457,11 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. highest-leverage thing in Step 4, and it is a MUST, not a nice-to-have: a PR-list call than omits `files` here is never corrected downstream — it just becomes one `gh pr view <n> --json files` per PR, run from inside the - `for pr in ...` loop this exact mistake produces. Measured on a real run: - the orchestrator listed PRs without `files` (5 `gh pr list` calls), then - looped `gh pr view --json files` once per PR to backfill it (9 calls) — - 100% avoidable had the first call carried `files`. There is no legitimate - reason to split these into two calls; `--json files` costs nothing extra - on the list call itself. + `for pr in ...` loop this exact mistake produces. This has happened on a + real run: the orchestrator listed PRs without `files`, then looped `gh pr + view --json files` once per PR to backfill it — entirely avoidable had the + first call carried `files`. There is no legitimate reason to split these + into two calls; `--json files` costs nothing extra on the list call itself. ```bash gh pr list -R <org>/<repo> --state merged --base <branch> \ @@ -470,8 +470,8 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. `--json files` returns every PR's changed paths in the SAME call, so one request per repo replaces one `gh pr view <n> --json files` per PR across - every coordinator. Measured across three real runs, per-PR file-list - fetches were **44 of 407 gh calls (10.8%)** — all of them avoidable here. + every coordinator. Across real runs, per-PR file-list fetches have been a + meaningful slice of all `gh` traffic — entirely avoidable here. Store the paths in each PR's `files` field rather than leaving it `null`: path-overlap is the first falsification test in `<pluginRoot>/skills/rca-build/references/github-evidence.md`, so with `files` populated a coordinator @@ -484,23 +484,23 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. pre-fetch's window) — never as a backfill for a PR-list call that should have carried `files` the first time. - Two other measured wastes this step should pre-empt: + Two other real wastes this step should pre-empt: - **Never let coordinators re-probe connectors.** `gh auth status` / - `kubectl version` accounted for **17 of 407 gh calls (4.2%)** purely + `kubectl version` calls have shown up repeatedly from coordinators purely because the manifest wasn't trusted. State plainly in the dispatch prompt that the gate validated them. - - **File contents were 31% of gh traffic** and are only partly predictable, - so do NOT bulk-fetch them. The `files` lists above tell a coordinator - exactly which files matter, and the tool cache dedupes the ones two - coordinators both open. + - **File contents are a large, only partly predictable slice of `gh` + traffic**, so do NOT bulk-fetch them. The `files` lists above tell a + coordinator exactly which files matter, and the tool cache dedupes the + ones two coordinators both open. 4. For each workload: run the connector skill's compulsory kubectl + VictoriaLogs sweep **once**, anchored to the build's own clock — never "now". **PAD the window: `started_at − 2m` .. `finished_at + 10m`.** `finished_at` is when the build was _marked_ finished, which is not when - the failing behaviour stopped: on one real build an upstream outage began - at 06:12:00 and ran to 06:15:51, while `finished_at` was 06:12:21 — a - sweep scoped strictly to `started_at..finished_at` saw 21 seconds of a - 4-minute outage and would have missed the cause entirely. Label every + the failing behaviour stopped: on a real build, an upstream outage was + still ongoing after `finished_at` was recorded — a sweep scoped strictly + to `started_at..finished_at` would have caught only the very start of it + and missed the cause entirely. Label every finding with whether it falls inside or outside the strict window so a coordinator can weigh it; do NOT silently widen to an arbitrary window (that is the separate, opposite failure of matching a coincidence from @@ -524,10 +524,10 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. note the weaker grounding — this note travels into the file, not just a spoken log line, so every coordinator sees it. 6. **Resolve local clones ONCE** (`lib/repo-source.mjs`). File *contents* are - the largest remaining slice of github traffic (31%), and most of it can be - served with no network at all when the machine already has the repos - checked out — measured `git show` ~37ms vs `gh api` ~1022ms for the same - file, byte-identical. + the largest remaining slice of github traffic, and most of it can be served + with no network at all when the machine already has the repos checked + out — a local `git show` returns the same bytes as `gh api` far faster, + with no round trip. ```js const d = discoverWorkspaceRoot({ repos: reposValidated, from: pluginRoot }); @@ -552,9 +552,9 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. like it worked. `pins` must be the **build-time commit shas** from `deployState`, never - branch names. A developer's clone is routinely stale (12 commits, measured), - and reading a branch locally returned different bytes than the real head — - for RCA that is a confident wrong answer about code that never shipped. + branch names. A developer's clone is routinely stale, and reading a branch + locally has returned different bytes than the real head — for RCA that is + a confident wrong answer about code that never shipped. Doing this at the gate is the point: every coordinator then reads a map instead of probing the filesystem itself. @@ -661,10 +661,10 @@ bookkeeping — no new tool calls needed for this part: `pre_seed` is already real evidence, not a guess. A representative still mid-loop (`NEEDS_INFO`/`PENDING`) has no `root_cause` yet — dispatching that cluster's siblings before it lands would degrade every one of them - into a full independent investigation (Step 5's measured cost note: 22.7 - tool calls/2.2 turns vs 8.0/2.0 for a representative, one run 60 - calls/17 minutes). Never do that; siblings of a not-yet-resolved - representative wait for Step 5 exactly as documented there. + into a full independent investigation, at real representative-level cost + instead of a cheap one-turn confirm (see Step 5's sibling-ordering note). + Never do that; siblings of a not-yet-resolved representative wait for + Step 5 exactly as documented there. - **NEEDS_INFO** → `recordTurn1(path, testRunId, {status: "NEEDS_INFO", threadId, asks}, nowMs)`. A real, non-terminal answer — hand it to Step 5's coordinator as `turn1_result` (never resubmit turn 1). @@ -732,11 +732,11 @@ per cluster, not global. A sibling is only cheap because it confirms a hypothesis someone else already established. Dispatch one without that hypothesis and "one-turn confirm" degenerates into a full independent investigation *with the sibling framing on -top*, so it costs MORE than the representative it was meant to be a fraction of. -Measured on a real run: siblings averaged **22.7 tool calls and 2.2 turns** -against **8.0 and 2.0** for the representative, and one burned **60 calls over -17 minutes**. Nothing ordered them after their rep and nothing refused to -dispatch without a seed, so it degraded silently. +top*, so it costs MORE than the representative it was meant to be a fraction +of — this has happened on a real run, with siblings running well past +representative-level cost because nothing ordered them after their rep and +nothing refused to dispatch without a seed. It degrades silently, with no +error to flag it. `siblingPreSeed` returns `{ok:false, reason}` when the representative is not resolved or recorded no `root_cause` — **do not dispatch that sibling yet**. @@ -835,8 +835,8 @@ closed). reference-doc / lib path.** A coordinator is dispatched fresh, with no guarantee about its own cwd — `references/evidence-routing.md` (bare, relative) resolves against whatever directory the coordinator happens to -start in, which is routinely NOT this plugin's root. Measured: 12 calls -across coordinators were `Read` attempts at the wrong bare path followed by a +start in, which is routinely NOT this plugin's root. This has cost real +coordinators repeated `Read` attempts at the wrong bare path followed by a `find` to recover the real one (`<pluginRoot>/skills/rca-build/references/evidence-routing.md`, `.../github-evidence.md`, `.../clustering.md`). Every dispatch prompt must state `pluginRoot=<absolute path>` up front and every reference-doc pointer in @@ -848,10 +848,9 @@ the coordinator re-derive it.** State plainly in the dispatch prompt: "Function signatures for `lib/*.mjs` are documented at `<pluginRoot>/skills/rca-build/SKILL.md` § API reference — read that section once if a signature is needed; do not `grep`/`Read`/`cat` the `lib/` source to re-derive a signature already -documented there." Measured: 15 of 49 self-discovery-tax calls were a -coordinator re-deriving a `lib/*.mjs` signature from source (one coordinator -read `lib/evidence-file.mjs` three times plus one `grep`, all to re-learn -`contributeLogsEvidence`'s signature) — a cost this pointer removes. +documented there." This is a real, recurring self-discovery tax — one +coordinator re-read `lib/evidence-file.mjs` plus a `grep`, all to re-learn +`contributeLogsEvidence`'s signature — a cost this pointer removes. **Coordinator prompts MUST name every connector-shaped skill on the manifest.** Each dispatch prompt lists, per capability, the resolved connector skill from @@ -895,20 +894,20 @@ result under the key it would compute — `mcpCacheKey(tool, args)` then `cachePut(toolCacheDirFor(buildId), key, {…, writerId: "orchestrator"}, nowMs)` from `lib/tool-cache.mjs` — storing the DIGEST, not the raw rows. -This is not optional polish; without it the MCP cache goes unused. Measured -across every live run before this was added: **zero MCP entries ever stored**, -because an agent's check-then-call-then-store costs three calls on a miss to -save one later, so skipping it is the rational choice for a one-off query. -Pre-seeding inverts that — the agent's `get` is a single call that usually -hits. Store the same digest you put in the evidence file; the two are -complementary (the file is read wholesale at turn 1, the cache answers a -specific repeat query later). +This is not optional polish; without it the MCP cache goes unused. Before this +was added, the cache went entirely unused across every live run — an agent's +check-then-call-then-store costs three calls on a miss to save one later, so +skipping it is the rational choice for a one-off query. Pre-seeding inverts +that — the agent's `get` is a single call that usually hits. Store the same +digest you put in the evidence file; the two are complementary (the file is +read wholesale at turn 1, the cache answers a specific repeat query later). **Also hand every dispatch the tool cache.** The evidence file shares digested _findings_; `bin/cached-exec.mjs` / `bin/cached-mcp.mjs` share raw _call -results_, which is where most duplicate work actually hides — on one measured -build `gh` was 37% of all coordinator tool calls and 46 were byte-identical -commands re-run by different coordinators. Include the plugin root in each +results_, which is where most duplicate work actually hides — `gh` calls make +up a large share of all coordinator tool calls on a real build, and a +meaningful number of them are byte-identical commands re-run by different +coordinators. Include the plugin root in each dispatch prompt so coordinators can invoke the wrappers, and tell them to pass their own `testRunId` as `writerId`. The cache lives at `<tmpdir>/bstack-rca/rca-toolcache.<buildId>/`, one file per call key, shared @@ -922,9 +921,9 @@ Step 4. Every coordinator writes only its own shard under open the same file for writing, concurrent write-back cannot lose an update; `readEvidenceFile` folds base + all shards into one view, applying shards in sorted order, with real evidence taking precedence over a recorded `gap`. A -measured comparison: 8 concurrent writers with a realistic read→work→write -window lost **28 of 40 updates** against a single shared file, and **0 of 40** -under this layout. +comparison under a realistic concurrent read→work→write window showed a +single shared file losing the large majority of concurrent updates, while +this sharded layout lost none. **Application bugs need a culprit PR.** Whenever a test's RCA classifies as PRODUCT_BUG / application bug, the coordinator MUST hunt the culprit PR via the From 164962f9d9b2b85c47324b7a9e5bf7ed55d7c934 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 21:51:46 +0530 Subject: [PATCH 46/51] perf(rca): enforce genuine parallel tool dispatch across gate, evidence pre-fetch, and coordinator probes Adds <use_parallel_tool_calls> guidance (per Anthropic's parallel-tool-use docs) to SKILL.md and ai-tfa-coordinator.md, bulletproofs Step 1's gate-probe batching rule (a real run violated the existing rule anyway - 4+ min cost), adds the same batching contract to Step 4's per-repo/per-workload pre-fetch loops and to github-evidence.md's per-probe github hunt, and adds tables of contents to evidence-routing.md/github-evidence.md (both >100 lines) per the agent-skills best-practices doc. --- agents/ai-tfa-coordinator.md | 24 ++++++++- skills/rca-build/SKILL.md | 53 ++++++++++++++++++- .../rca-build/references/evidence-routing.md | 7 +++ .../rca-build/references/github-evidence.md | 33 ++++++++++++ 4 files changed, 115 insertions(+), 2 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index d92e88c..0c8b853 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -29,6 +29,20 @@ standalone, driven by the batch workflow, a subagent dispatch, or the thin sequential harness (`lib/loop.mjs`). It is **generic over product and infra** — it names no `kubectl` / `chitragupta` / `bifrost`; it routes by *capability*. +<use_parallel_tool_calls> +For maximum efficiency, whenever you need to perform multiple independent +operations, invoke all relevant tools simultaneously rather than sequentially. +Prioritize calling tools in parallel whenever possible. For example, when +checking commit history across several candidate files, run all those `gh` +calls in parallel. When validating multiple connectors (github, infra, logs, +metrics) or their scope probes, run all of those checks in parallel. When a +NEEDS_INFO turn carries multiple asks, gather all of them in parallel. Err on +the side of maximizing parallel tool calls rather than running too many tools +sequentially — a real run measured 60-90 seconds of pure overhead per +avoidable sequential call. The only exception is when one call's output is a +literal input to another call; that pair, and only that pair, runs in order. +</use_parallel_tool_calls> + ## Inputs - `pluginRoot` — **required**, absolute path to this plugin's repo root. Every @@ -379,7 +393,15 @@ capability is unavailable — emit an turn — never one ask's full gather-and-digest before starting the next. `lib/loop.mjs`'s `runRcaLoop` mirrors this with `Promise.all` over `buckets.gather`; do the equivalent here. Only the final message assembly - respects priority order, not the fetching. + respects priority order, not the fetching. **This is not only an + across-asks rule** — a single github ask routinely needs several + independent probes itself (a commit-history check per candidate file, a + falsification check per suspect PR); see + `references/github-evidence.md`'s "Batch every independent probe into + one message" for that one-level-down case. One Bash call per message, + waiting for each result before firing the next independent probe, pays + a full turn's think-time per call for no reason — this was measured + costing 60-90s of pure overhead per call in a real run. For each ask: skip → record in asks_skipped, emit nothing. gather → FIRST check `evidenceFile` (if present) for this ask's scope — diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index ca21852..7358321 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -26,6 +26,21 @@ There is exactly **one mode**: autonomous. There is exactly **one gate** (Step Config (concurrency, turn-cap, paths, evidence registry) lives in `config/rca.config.json`. State lives in the CSV/WAL spine (`lib/csv-state.mjs`). +<use_parallel_tool_calls> +For maximum efficiency, whenever you need to perform multiple independent +operations — connector probes, per-repo evidence fetches, per-workload log +sweeps, or any other set of calls with no dependency between them — invoke all +relevant tools simultaneously in one message rather than sequentially. +Prioritize calling tools in parallel whenever possible; err on the side of +maximizing parallel tool calls rather than running too many tools +sequentially. This applies throughout every step below (Gate probes, Step 4's +per-repo/per-workload pre-fetch, Step 4b's cluster dispatch, Step 5's +representative and sibling dispatch) — a real run measured this exact +violation costing 4+ minutes on gate probes alone. The only exception is when +one call's output is a literal input to another; that pair, and only that +pair, runs in order. +</use_parallel_tool_calls> + ## API reference — read THIS, do not grep the source Every signature this run needs, in one place. This exists because agents were @@ -217,6 +232,30 @@ sequential-*looking* list of independent checks got executed sequentially in practice, costing minutes it never needed to. Don't repeat that here, at the very front of the pipeline where it delays everything downstream.) +**This rule has already been read and violated on a real run — measured +cost 4+ minutes on gate/scope probes alone.** The coordinator had this exact +paragraph available and still issued `gh api <repo>`, an env-var check, a +second env-var check, `kubectl get ns`, `kubectl auth can-i`, `kubectl get +pods` (×2), and `kubectl get pod` as eight separate messages, one Bash call +each, 10-34 seconds apart. Restating the rule again clearly did not prevent +that, so treat it as a hard gate, not a preference: + +- **REQUIRED before your first probe Bash call:** write out the full list of + every probe you are about to run this pass — every base probe, every scope + probe, every target — one line each. Then issue every item on that list as + its own tool-call block **in this one message**. +- **If a message you are about to send contains exactly one Bash call for a + probe, and your list above still has unissued items with no dependency on + that call's result — STOP.** That message is the violation in progress. + Add the rest of the list to it before sending. +- "I'll check github's connector first, then move to infra" is the + rationalization that produced the 4-minute real-run cost above. It sounds + like reasonable sequencing; it is the forbidden pattern. github's probes + and infra's probes have no dependency on each other — there is no "first." +- The only real dependency is per-connector: a connector's scope probes wait + on that SAME connector's base probe, nothing else. Two different + connectors' probes never wait on each other, ever. + | Connector | Probe | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | github | `gh auth status` (or a GitHub MCP tool listed) | @@ -452,6 +491,15 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. A repo the connector can't reach records `{gap: "<reason>"}` — never blocks the rest of the pre-fetch. + **Every repo's PR-window search is independent of every other repo's — + fire all of them as parallel tool calls in ONE message, never one repo, + read its result, then the next repo.** The same rule that governs Gate + Part A's connector probes applies here at repo granularity: write the + full repo list from step 2 first, then issue every repo's `gh pr list` + call together. A message containing exactly one repo's fetch, with other + repos from the union still unfetched and no dependency on this one's + result, is the violation — go back and batch the rest in before sending. + **`--json` on THIS FIRST `gh pr list` call MUST include `files` — there is no separate step where it gets added later.** This is the single highest-leverage thing in Step 4, and it is a MUST, not a nice-to-have: a @@ -495,7 +543,10 @@ evidenceType, fn)` to dedupe if two steps need the same `(repo, range)`. ones two coordinators both open. 4. For each workload: run the connector skill's compulsory kubectl + VictoriaLogs sweep **once**, anchored to the build's own clock — never - "now". **PAD the window: `started_at − 2m` .. `finished_at + 10m`.** + "now". **Every workload's sweep is independent of every other workload's + and of every repo's fetch in step 3 — batch all of them into the same + message(s), same rule as step 3's repo fetches.** **PAD the window: + `started_at − 2m` .. `finished_at + 10m`.** `finished_at` is when the build was _marked_ finished, which is not when the failing behaviour stopped: on a real build, an upstream outage was still ongoing after `finished_at` was recorded — a sweep scoped strictly diff --git a/skills/rca-build/references/evidence-routing.md b/skills/rca-build/references/evidence-routing.md index 9132fef..2c3ba5b 100644 --- a/skills/rca-build/references/evidence-routing.md +++ b/skills/rca-build/references/evidence-routing.md @@ -13,6 +13,13 @@ capability manifest — see `SKILL.md` § Gate Part A). There are **no `kubectl` `chitragupta` / `bifrost` literals here** — that is the whole point of going generic. +**Contents:** [How asks are processed](#how-a-turns-asks-are-processed) · +[Routing table](#routing-table-capability-not-tool) · +[Digest format](#digest-format) · +[Unfulfillable asks](#unfulfillable-asks--report-dont-drop) · +[Capability manifest](#capability-manifest-built-once-at-the-gate) · +[Build-level evidence cache](#build-level-evidence-cache-compute-once) + The registry logic lives in `lib/routing.mjs` (`routeAsk` / `routeAsks`); this file is the human/agent-facing contract for the digest and the size caps. diff --git a/skills/rca-build/references/github-evidence.md b/skills/rca-build/references/github-evidence.md index a3d6b3f..2960a9c 100644 --- a/skills/rca-build/references/github-evidence.md +++ b/skills/rca-build/references/github-evidence.md @@ -9,6 +9,15 @@ that tries to *disprove* each suspect before it enters `related_prs`. > needed and use whatever the client already has — **GitHub MCP if available, > else `gh`, else degrade** to an `unavailable` block. +**Contents:** [Capability discovery](#capability-discovery-in-order) · +[Culprit-PR hunt](#application-bugs-require-a-culprit-pr-hunt-mandatory) · +[Batching probes](#batch-every-independent-probe-into-one-message--never-one-call-per-turn) · +[Evidence per ask](#evidence-each-ask-needs-be-specific--no-fishing) · +[Field-filtering](#field-filtering--project-before-you-pull-every-call) · +[Falsification protocol](#falsification-protocol--rule-out-dont-just-rule-in) · +[Suspect packet](#the-suspect-packet-structured-not-free-text) · +[Digest discipline](#digest-discipline) + ## Capability discovery (in order) 1. **GitHub MCP** (`mcp__github__*`) — preferred for structured PR/diff/blame queries. @@ -41,6 +50,30 @@ the CSV row records the gap. Never fabricate a PR; if the github connector is invalid/absent, the same explicit statement plus an `unavailable` block goes to TFA (a gate-recorded gap). +## Batch every independent probe into one message — never one call per turn + +This hunt routinely needs several `gh` calls that don't depend on each +other's output: a commit-history check per candidate file in "changed paths +vs failure signature," each row of the "Evidence each ask needs" table +below, and each candidate PR's falsification check. **None of these need to +see a prior result before running** — the only exception is when one call's +output supplies a literal input to the next (e.g., you need a PR number +back from a search before you can `gh pr view` it). + +Issue every independent probe as its own tool call **within the same +message** — the same discipline `ai-tfa-coordinator.md`'s NEEDS_INFO step +already requires across multiple asks (`Promise.all` / concurrent gather) +applies here too, one level down, across multiple probes inside a single +ask. One call per file path, fired one message at a time, waiting for each +result before issuing the next, spends a full turn's think-time on every +individual `gh api` round trip even though the call itself finishes in +under two seconds — for a five-file changed-paths check that is the +difference between one batched message and five serialized ones. Plan the +full probe list first (every candidate file, every table row, every +falsification check that has no dependency on another probe's result), then +fire all of them together; only serialize the ones with a genuine +input-from-output dependency. + ## Evidence each ask needs (be specific — no fishing) | Ask intent | Gather exactly | From 395960c2d7ff50efa958c3a63c6f29fb2ea885ac Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 22:32:36 +0530 Subject: [PATCH 47/51] fix(rca): bulletproof 4 real step-skips found in a production run A forensic pass on a real run (build hazbi3bs...) found four documented steps were skipped entirely, not just executed out of order: - Step 1A: only one of three present connector families was ever opened (a11y-regression-context, the one matching the build's actual failures, was never read) - adds a required enumerate-then-match check. - Step 1B: two separate AskUserQuestion calls instead of one, for fields the selected connector's own intake-defaults section already answered outright - adds a check-the-connector-first rule and bulletproofs "never a second question" with a STOP condition. - Step 1A scope probes: zero scope-probe calls ran despite the connector declaring seven - adds a required pre-Step-2 checkpoint. - Step 3: clusterAndPersist fallback ran without ever attempting the preferred getBuildFailureThemes call first - adds a required-order gate. - Step 5: fired the full fan-out with Step 4b (turn-1 pre-dispatch) never having run at all, so every cluster paid full coordinator cost with none of the one-turn resolve-and-skip fast path attempted - adds a required Step-4b-happened gate before Step 5's first dispatch. --- skills/rca-build/SKILL.md | 84 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 2 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index 7358321..cd5e78d 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -187,6 +187,23 @@ product-scoped — a workspace may hold none, one, or several product families (e.g. `<product-a>-*`, `<product-b>-*`, whatever the user has). After the `ls`, pick the _product family_ whose connector skills apply to THIS build: +**REQUIRED before you open any single family's SKILL.md: list every family the +`ls` output actually returned, one line each, THEN check each one's failure- +signature match — never open just the first (or only) one you happen to +notice and stop there.** This has gone wrong on a real run: the `ls` returned +three families (`a11y-*`, `tm-*`, `tra-*`), the build's actual failures were +accessibility/Workflow-Analyzer domain, and the orchestrator read only +`tra-regression-context` (a TRA/Observability connector whose declared lanes +don't include accessibility at all) — never opened `a11y-regression-context`, +the one that actually matched. That silently produced "exactly one family, use +it" behavior even though three were present, and the mismatch then had to be +patched by asking the user two separate questions Part B says never to ask. +**If you are about to read one family's SKILL.md and cannot recite the other +families the `ls` output also returned, STOP — you skipped the enumeration.** +The failure-signature check (step 2 below) is what catches a family that looks +present but doesn't actually own this build's failures; skipping straight to +one file is exactly how a wrong-family read reaches Part B undetected. + - **Zero families found** → **nudge the user in the gate summary**: "No connector-shaped skills found under `.claude/skills/` — proceeding with raw MCP tools only; culprit-PR attribution will be best-effort against @@ -294,6 +311,17 @@ Coordinators can then act freely inside the resolved scope and must fail closed outside it. This closes the failure mode where a coordinator degrades to `unavailable` because the orchestrator didn't confirm the specific target. +**A real run skipped this whole section — not one scope probe ran, for a +connector that declares seven of them.** The base probes (`gh auth status`, +`kubectl version`) passed and the run went straight to Step 2's `listTestIds`, +never reading or running the connector's `Scope probes:` list at all. **Before +your first `listTestIds`/discovery call: confirm you can name every scope +probe you ran and its result, for every connector recorded `valid` in the +manifest.** If a connector is `valid` in the manifest and you cannot name a +single scope-probe result for it, STOP — go back and run its declared list (or, +if it genuinely declares none, the manifest-time warning below is the only +legitimate reason to have nothing to name). + Skills that don't declare `Scope probes:` degrade to a manifest-time warning ("scope probes missing — coordinator may over-degrade"). Do not invent product-specific probes here. @@ -316,6 +344,21 @@ is the point: - the current branch for the working branch, - cheap inference (e.g. the automation repo is the cwd if it holds the tests). +**Check the selected connector skill's own intake-defaults section FIRST — before +falling through to inference, and before ever asking.** A connector skill that +declares "Intake defaults for the gate (Part B)" (or equivalent) is telling you +these fields are answerable outright for its product, by build-name/lane or +failure-pattern lookup — not assumptions, not something to ask about. Skipping +straight to inference or to the user when the connector already names the +answer is the exact bug a real run hit: the selected connector's own intake +section explicitly read _"An orchestrator that... asks the consolidated +question about them is reading the wrong place"_, and the run asked anyway — +two separate questions, not even the allowed single one. If the connector's +intake section doesn't resolve a field for THIS build (e.g. its lane table +doesn't match the failure signature at all), that is itself a sign the wrong +family was selected — go back to the enumeration step above before treating +the field as genuinely non-assumable. + **Product-repo corroboration (do NOT skip).** The product repo must plausibly be the _system under test for THIS build's failures_ — not merely a repo name found lying around. A repo mentioned only in workspace docs/READMEs is a **weak @@ -347,7 +390,20 @@ above — without it the culprit-PR hunt cannot run); and rarely an ambiguous re when PRs were supplied. Those, and only those, may be asked **ONCE, in a single consolidated question at gate close** — e.g. _"Failures look like `<domain>`; which repo owns that code? (reply 'none' → I'll RCA without culprit-PR -attribution)."_ Never a second question. **Headless: skip asking entirely; +attribution)."_ Never a second question. + +**This has already been violated on a real run** — a product-family +disambiguation question and a repo-ownership question went out as two separate +`AskUserQuestion` calls, 33 seconds apart, instead of one consolidated +question (or, better, no question at all, since the connector's own intake +defaults answered both — see above). **Before your first `AskUserQuestion` +call this pass: write out every field this run still needs from the user, +across every reason it might be non-assumable, in one list — then ask them as +ONE question with multiple parts if more than one survives.** If you are about +to send a second `AskUserQuestion` call in the same gate pass, STOP — fold its +content into the first question instead, or if the first has already been +sent, that is the violation; there is no second gate question, ever. +**Headless: skip asking entirely; record the gaps.** ### Gate close @@ -404,7 +460,17 @@ Each cluster gets one **representative** (full multi-turn loop) and `N−1` the expensive evidence hunt to O(distinct causes) while every test still lands a per-test RCA. Singleton clusters are just plain per-test loops. -**Prefer the server's own clustering over recomputing it client-side:** +**Prefer the server's own clustering over recomputing it client-side.** A real +run skipped straight to the client-side fallback below without ever calling +`getBuildFailureThemes` — the tool's schema had even been loaded via +`ToolSearch` that pass, it was simply never invoked. **`clusterAndPersist` may +ONLY be called after a `getBuildFailureThemes` call this pass returned +`ready: false` (or errored) — never as a first move.** If you are about to call +`clusterAndPersist` and cannot point to this pass's own `getBuildFailureThemes` +call and its `ready: false` result, STOP — you are taking the fallback without +ever having tried the preferred path, which throws away the server's own +root-cause grouping for no reason and degrades every run to text-signature +clustering by default instead of by necessity. 1. Call `getBuildFailureThemes(buildUuid=<build id>)`. If nothing has ever been computed for this build, this triggers computation (one POST, same @@ -773,6 +839,20 @@ representative outcome for seeding siblings. ## Step 5 — fan-out (fully autonomous) +**REQUIRED gate before your first Step 5 dispatch: Step 4b must have already +happened this pass.** A real run skipped Step 4b entirely — no lightweight +turn-1 pre-dispatch subagent was ever launched, and all N cluster +representatives went straight to a full `ai-tfa-coordinator` dispatch here +instead, paying full multi-turn coordinator cost for every cluster including +the ones that would have resolved in one pre-dispatched turn. **If you are +about to issue Step 5's representative dispatches and cannot point to this +pass's `initTurn1Registry` call and a turn-1 dispatch batch issued for every +thread-less cluster representative, STOP — go back and do Step 4b first, even +belatedly.** Step 4b is not an optional latency nicety layered on top of Step +5; skipping it means every single cluster pays for a capability (Step 4b's +one-turn resolve-and-skip-Step-5 fast path) that this run never even +attempted. + **ORDER MATTERS: representative first, siblings only after it lands.** For each cluster, dispatch the representative, wait for its row to go terminal, then dispatch its siblings carrying `pre_seed` from From dafcd291f3d8a4e504e4a7e1f49500f22f063b34 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 22:50:22 +0530 Subject: [PATCH 48/51] fix(rca): clarify Step 5's Step-4b gate checks dispatch, not completion The prior wording ("Step 4b must have already happened", "go back and do Step 4b first") could be misread as waiting for Step 4b's subagents to finish before Step 5 starts - reintroducing the sequential-latency bug Step 4b exists to remove. Reworded to be explicit: the gate only checks that the dispatch batch was issued (fire-and-forget), same contract Step 4b already documents, not that it completed. --- skills/rca-build/SKILL.md | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index cd5e78d..d76587c 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -839,19 +839,24 @@ representative outcome for seeding siblings. ## Step 5 — fan-out (fully autonomous) -**REQUIRED gate before your first Step 5 dispatch: Step 4b must have already -happened this pass.** A real run skipped Step 4b entirely — no lightweight -turn-1 pre-dispatch subagent was ever launched, and all N cluster -representatives went straight to a full `ai-tfa-coordinator` dispatch here -instead, paying full multi-turn coordinator cost for every cluster including -the ones that would have resolved in one pre-dispatched turn. **If you are -about to issue Step 5's representative dispatches and cannot point to this -pass's `initTurn1Registry` call and a turn-1 dispatch batch issued for every -thread-less cluster representative, STOP — go back and do Step 4b first, even -belatedly.** Step 4b is not an optional latency nicety layered on top of Step -5; skipping it means every single cluster pays for a capability (Step 4b's -one-turn resolve-and-skip-Step-5 fast path) that this run never even -attempted. +**REQUIRED gate before your first Step 5 dispatch: Step 4b's dispatch batch +must have already been ISSUED this pass — not completed, not waited on, +issued.** A real run skipped Step 4b entirely — no lightweight turn-1 +pre-dispatch subagent was ever launched, and all N cluster representatives +went straight to a full `ai-tfa-coordinator` dispatch here instead, paying +full multi-turn coordinator cost for every cluster including the ones that +would have resolved in one pre-dispatched turn. **If you are about to issue +Step 5's representative dispatches and cannot point to this pass's +`initTurn1Registry` call and a turn-1 dispatch batch issued for every +thread-less cluster representative, STOP — go back and fire that dispatch +batch first.** This gate is about the dispatch having gone out, same +fire-and-forget contract Step 4b already documents — it is NOT a "wait for +Step 4b's subagents to finish" gate, and reading it that way reintroduces the +exact sequential-latency bug Step 4b exists to remove. In practice this batch +should already be long since fired by the time you reach Step 5, since Step +4b's own instructions have it go out in the same turn as Step 4's first +evidence-gathering calls — this check exists only to catch the case where +that never happened at all, not to insert a new wait. **ORDER MATTERS: representative first, siblings only after it lands.** For each cluster, dispatch the representative, wait for its row to go terminal, then From b726aa103ab545018bdf192595afb7ff820d9067 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Tue, 4 Aug 2026 23:09:52 +0530 Subject: [PATCH 49/51] fix(rca): pin clustersFromThemes' rows source to the seeded CSV, not a stale variable Traced a real "test id mismatch -> falls back to client-side clustering" report to actual data: getBuildFailureThemes/listTestsInFailureTheme both returned correct, matching numeric testRunIds - no format bug. But the run's final CSV carried clusterAndPersist's c-xxxxx signature-hash IDs, not clustersFromThemes' theme-<id>/solo-<id> format, meaning the preferred path was silently abandoned. Root cause: an earlier listTestIds(status=failed) call had errored once and a later call used a different status filter in the same session - clustersFromThemes was fed whatever `rows` variable was still in scope, so every theme member came back unmatched (looks exactly like an ID mismatch, isn't one). Pins rows to readRows(csvPath) - the one row set guaranteed fresh and from a successful seed. --- skills/rca-build/SKILL.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/skills/rca-build/SKILL.md b/skills/rca-build/SKILL.md index d76587c..2c4be5e 100644 --- a/skills/rca-build/SKILL.md +++ b/skills/rca-build/SKILL.md @@ -479,14 +479,30 @@ clustering by default instead of by necessity. 2. **`ready: true`** → for each entry in `buildThemes`, call `listTestsInFailureTheme(buildUuid=<build id>, themeId=<buildFailureThemeId>)`, following `nextCursor` until exhausted, to get that theme's member - testRunIds. Feed the `listTestIds` rows + the themes result + the per-theme - member lists into `lib/theme-clustering.mjs` → `clustersFromThemes(rows, -themesResult, testsByThemeId)` — this is the **preferred path**, since the - grouping reflects the server's own root-cause analysis rather than a text- - signature guess, and it never runs the coordinator fan-out N-tests-wide for - a build with only a handful of distinct causes duplicated across teams. - Any failed test the server didn't assign to a theme is never dropped — it - still gets its own singleton cluster. + testRunIds. Feed rows + the themes result + the per-theme member lists into + `lib/theme-clustering.mjs` → `clustersFromThemes(rows, themesResult, + testsByThemeId)` — this is the **preferred path**, since the grouping + reflects the server's own root-cause analysis rather than a text-signature + guess, and it never runs the coordinator fan-out N-tests-wide for a build + with only a handful of distinct causes duplicated across teams. Any failed + test the server didn't assign to a theme is never dropped — it still gets + its own singleton cluster. + + **`rows` MUST be `readRows(csvPath)` — the CSV Step 2 already seeded — + never a `listTestIds` result variable held over from earlier in the turn.** + A real run hit exactly this: an earlier `listTestIds(status="failed")` call + in the same session errored ("fetch failed"), a later call used a + *different* status filter, and `clustersFromThemes` was fed whatever `rows` + was still in scope — every theme member came back unmatched (every + `rowById.get(...)` lookup missed), which reads exactly like a "test ID + mismatch" but isn't one: `getBuildFailureThemes`/`listTestsInFailureTheme` + themselves returned correct data the whole time. The result: the CSV ended + up with signature-hash `c-xxxxx` cluster IDs (`clusterAndPersist`'s fallback + format) instead of `theme-<id>`/`solo-<id>`, i.e. the preferred path was + silently abandoned even though it never actually failed. The CSV is the one + row set guaranteed fresh and from a successful seed (Step 2 only seeds + after `listTestIds` succeeds) — always re-read it here rather than trusting + a variable carried over from turns ago. 3. **`ready: false`** (still computing past the poll budget, a failure status, or `status: "trigger-unavailable"` — the trigger call didn't succeed) → **fall back** to **`clusterAndPersist(csvPath, csvStateModule)`** From 24d81799574c3d5db90c99397143e39d57353a9e Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Wed, 5 Aug 2026 00:25:24 +0530 Subject: [PATCH 50/51] fix(rca): stop resubmitting a thread after 2 consecutive TFA wedges, not turnCap Confirmed via real production logs (test-failure-agent-chat, namespace testcasegeneration): a thread that gets "TFA agent run failed" twice in a row is hitting the backend's openai.BadRequestError context_length_exceeded, not a transient wedge - the thread's accumulated history exceeded the model's context window, which a resubmit cannot fix since history doesn't shrink. Rule 4b previously advised resubmitting all the way to turnCap, wasting every remaining turn on a thread that structurally cannot recover. Now: one retry for a first failure (still handles genuine transient wedges), but two consecutive same-thread failures end the test PENDING immediately with a new distinct note ("likely-context-exceeded"), added as a 4th documented status alongside turn-cap/soft-pending/blocked. --- agents/ai-tfa-coordinator.md | 52 +++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 13 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index 0c8b853..f5e1d2c 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -230,19 +230,40 @@ read-only and has no side effects, so a read is always safe to repeat. extra turn, never a busy-wait. 4. **One thread per test.** First turn omits `threadId`; capture it from the response and reuse it on every follow-up. Never start a second thread. -4b. **A drain ERROR kills the TURN, not the THREAD — resubmit, don't give up.** +4b. **A drain ERROR kills the TURN, not always the THREAD — resubmit ONCE, + don't give up immediately, but don't retry blindly to the turn cap either.** `getTfaTurnResult` returning `TFA agent run failed` (or the submit itself - throwing it) is a dead turn, not a dead thread: observed repeatedly, a - fresh submit on the SAME `threadId` succeeds immediately and resolves at - high confidence. So when the drain fast-fails on consecutive hard errors, - the next move is to resubmit on that same thread (counting it as a turn) — - NOT to mint a new thread and not to end the run `PENDING`. Ending PENDING - here throws away a resolvable test. Only stop once the turn cap is spent. + throwing it) is usually a dead turn, not a dead thread: a fresh submit on + the SAME `threadId` frequently succeeds immediately and resolves at high + confidence. So on the FIRST such failure, resubmit on that same thread + (counting it as a turn) — do NOT mint a new thread and do NOT end the run + `PENDING` on one failure alone. + + **But if THAT resubmit ALSO comes back `TFA agent run failed` — two + consecutive failures on the same thread with no successful real response + between them — STOP retrying and end the test `PENDING` immediately (note + `"likely-context-exceeded"`), rather than continuing to resubmit until the + turn cap is spent.** This is confirmed, not a guess: real production logs + for a thread that wedged this way twice in a row show the backend's own + error was `openai.BadRequestError: ... 'code': 'context_length_exceeded'` + both times — the thread's accumulated history had exceeded the model's + context window, a structural condition that does NOT clear on resubmit + (unlike a genuinely transient wedge, which the first retry already + handles). Burning the remaining turn cap on more resubmits of a thread in + this state wastes every one of them — they cannot succeed. Two consecutive + failures on the same thread (not two failures total across the whole run) + is the signal; a thread that fails once, then succeeds, then fails again + later has NOT hit this — treat each failure's retry independently and only + escalate to this stop when a resubmit's own retry fails too. 4b-i. **Two DIFFERENT TFA failures, don't confuse them.** - - `TFA agent run failed` — the wedge. Unrelated to message size (a - 240-char message wedged like a 1500-char one). Fix: resubmit on the same - thread, per 4b. + - `TFA agent run failed` — usually the wedge (see 4b: one retry). Two of + these in a row on the same thread is very likely `context_length_exceeded` + server-side (confirmed via production logs, not inferred) — stop per 4b + rather than treating it as a message-size problem to fix by shortening + THIS turn's submission; the accumulated thread history, not this + message, is what's oversized, and this coordinator has no way to trim + that from the client side. - **`turnId` exists ONLY on a soft-`PENDING` turn.** TFA returns `{status, threadId, turnId}` for PENDING and omits `turnId` entirely on `RESOLVED` / `NEEDS_INFO` — so reporting `turn_id: not available` on a resolved @@ -419,6 +440,10 @@ capability is unavailable — emit an PRODUCT_BUG in play + no supported PR yet → widen the github hunt this turn. Concatenate per-ask blocks into the next-turn MESSAGE (respect size caps). 4. SUBMIT follow-up on the SAME thread: tfaRcaTurn(testRunId, message, threadId). turns_used += 1. + FAILS ("TFA agent run failed") → resubmit the SAME message on the SAME + thread once (per 4b), still counting as a turn. If THAT resubmit also + fails → END (PENDING, note "likely-context-exceeded") immediately, do + NOT continue looping to the turn cap (per 4b/4b-i). 5. TURN-CAP CHECK: if turns_used >= turnCap and still NEEDS_INFO → END (PENDING, "turn-cap"). else → go to 2 with the new result. 6. EMIT the RCA_OUTPUT block from the captured terminal state. @@ -492,9 +517,10 @@ RCA_OUTPUT_END Notes: - `status` is one of exactly three values. `turn-cap`, `soft-pending` (drain - budget spent) and `blocked` all report as `PENDING`; note which in `root_cause`. - A `PENDING` from a *drained* turn should never appear — a drain that lands - re-classifies instead. + budget spent), `blocked`, and `likely-context-exceeded` (two consecutive + same-thread `TFA agent run failed` resubmits, per 4b) all report as + `PENDING`; note which in `root_cause`. A `PENDING` from a *drained* turn + should never appear — a drain that lands re-classifies instead. - `asks_skipped` always includes `test_logs` whenever TFA asked for logs. `asks_fulfilled` **never** includes `test_logs`. - `asks_unavailable` is the evidence-coverage signal the coverage stamp turns From ef8d695df53d2038f27576e219e0adfcbcfbf1e5 Mon Sep 17 00:00:00 2001 From: Jaydeep Dave <davejaydeep48@gmail.com> Date: Wed, 5 Aug 2026 00:32:22 +0530 Subject: [PATCH 51/51] fix(rca): recover from context-exceeded via one condensed-hypothesis thread restart, not just a faster PENDING The prior fix (end PENDING after 2 consecutive same-thread wedges) only failed cleaner - it didn't get the test to an actual resolution, which is the real goal. Since the dead thread's oversized accumulated history is the only broken part (not the test itself), add a real recovery: on confirmed context_length_exceeded (2 consecutive same-thread failures), distill everything gathered so far into one condensed hypothesis message and submit it as turn 1 of a brand new thread - a single, narrow, justified exception to "never start a second thread" in step 4, since the first thread is provably dead rather than merely inconvenient. Bounded to exactly one restart per test; a restarted thread that also overflows twice ends PENDING for real rather than restarting indefinitely. --- agents/ai-tfa-coordinator.md | 92 ++++++++++++++++++++++-------------- 1 file changed, 57 insertions(+), 35 deletions(-) diff --git a/agents/ai-tfa-coordinator.md b/agents/ai-tfa-coordinator.md index f5e1d2c..f0284af 100644 --- a/agents/ai-tfa-coordinator.md +++ b/agents/ai-tfa-coordinator.md @@ -228,42 +228,60 @@ read-only and has no side effects, so a read is always safe to repeat. 3. **Turn-cap** = `turnCap` from `config/rca.config.json` (default 6). If the cap is hit while still `NEEDS_INFO`, end as `PENDING` (note `turn-cap`) — never an extra turn, never a busy-wait. -4. **One thread per test.** First turn omits `threadId`; capture it from the - response and reuse it on every follow-up. Never start a second thread. -4b. **A drain ERROR kills the TURN, not always the THREAD — resubmit ONCE, - don't give up immediately, but don't retry blindly to the turn cap either.** - `getTfaTurnResult` returning `TFA agent run failed` (or the submit itself - throwing it) is usually a dead turn, not a dead thread: a fresh submit on - the SAME `threadId` frequently succeeds immediately and resolves at high - confidence. So on the FIRST such failure, resubmit on that same thread - (counting it as a turn) — do NOT mint a new thread and do NOT end the run - `PENDING` on one failure alone. - - **But if THAT resubmit ALSO comes back `TFA agent run failed` — two +4. **One thread per test — with one narrow, deliberate exception (4b).** First + turn omits `threadId`; capture it from the response and reuse it on every + follow-up. Never start a second thread EXCEPT the single context-exceeded + restart 4b describes — that path exists precisely because the first + thread is provably unrecoverable, not as a general license to abandon + threads that are merely inconvenient. +4b. **A drain ERROR kills the TURN, not always the THREAD — resubmit ONCE; if + that ALSO fails, RESTART with a condensed hypothesis rather than just + giving up.** `getTfaTurnResult` returning `TFA agent run failed` (or the + submit itself throwing it) is usually a dead turn, not a dead thread: a + fresh submit on the SAME `threadId` frequently succeeds immediately and + resolves at high confidence. So on the FIRST such failure, resubmit on + that same thread (counting it as a turn) — do NOT mint a new thread and do + NOT end the run `PENDING` on one failure alone. + + **If THAT resubmit ALSO comes back `TFA agent run failed` — two consecutive failures on the same thread with no successful real response - between them — STOP retrying and end the test `PENDING` immediately (note - `"likely-context-exceeded"`), rather than continuing to resubmit until the - turn cap is spent.** This is confirmed, not a guess: real production logs - for a thread that wedged this way twice in a row show the backend's own - error was `openai.BadRequestError: ... 'code': 'context_length_exceeded'` - both times — the thread's accumulated history had exceeded the model's - context window, a structural condition that does NOT clear on resubmit - (unlike a genuinely transient wedge, which the first retry already - handles). Burning the remaining turn cap on more resubmits of a thread in - this state wastes every one of them — they cannot succeed. Two consecutive - failures on the same thread (not two failures total across the whole run) - is the signal; a thread that fails once, then succeeds, then fails again - later has NOT hit this — treat each failure's retry independently and only - escalate to this stop when a resubmit's own retry fails too. + between them — this is confirmed (via real production logs, not a guess) + to be the backend's own `openai.BadRequestError: ... + 'code': 'context_length_exceeded'`: the thread's accumulated history has + exceeded the model's context window, a structural condition that does NOT + clear on resubmit (unlike a genuinely transient wedge, which the first + retry already handles). Continuing to resubmit THIS thread wastes every + remaining turn — none can succeed. But the test itself is very likely + still resolvable; only this one thread's history is oversized. So:** + + 1. **Distill everything gathered so far this run into ONE condensed + hypothesis message** — same digest discipline as everywhere else (link + over paste, no raw diffs/log dumps): the leading root-cause hypothesis, + the strongest supporting evidence, and any suspect PR, in the same + shape a cluster sibling's `pre_seed` message would carry. Discard the + rest of the dead thread's history entirely — it is exactly what caused + the overflow, so carrying more of it into the restart than this one + condensed paragraph defeats the point. + 2. **Submit this as turn 1 of a BRAND NEW thread** (`tfaRcaTurn(testRunId, + message=<condensed hypothesis>)`, no `threadId`) — this is the one + narrow exception to "never start a second thread" in step 4, justified + because the first thread is now provably dead, not merely difficult. + Capture the new `threadId` and continue the loop from step 2 as normal; + its turns count against the same overall `turnCap` — no separate budget. + 3. **Allow exactly ONE such restart per test.** If the fresh thread ALSO + hits two consecutive same-thread failures, do not restart again — end + `PENDING` (note `"likely-context-exceeded"`) for real. A test whose + condensed restart still overflows needs a human, not a third thread. 4b-i. **Two DIFFERENT TFA failures, don't confuse them.** - - `TFA agent run failed` — usually the wedge (see 4b: one retry). Two of - these in a row on the same thread is very likely `context_length_exceeded` - server-side (confirmed via production logs, not inferred) — stop per 4b - rather than treating it as a message-size problem to fix by shortening - THIS turn's submission; the accumulated thread history, not this - message, is what's oversized, and this coordinator has no way to trim - that from the client side. + - `TFA agent run failed` — usually the wedge (see 4b: one retry, then one + condensed restart if the retry also fails). Two of these in a row on the + same thread is very likely `context_length_exceeded` server-side + (confirmed via production logs, not inferred) — handle per 4b rather + than treating it as a message-size problem to fix by shortening THIS + turn's submission; the accumulated thread history, not this message, is + what's oversized, and a same-thread resubmit can never trim that — only + a fresh thread with a condensed message can. - **`turnId` exists ONLY on a soft-`PENDING` turn.** TFA returns `{status, threadId, turnId}` for PENDING and omits `turnId` entirely on `RESOLVED` / `NEEDS_INFO` — so reporting `turn_id: not available` on a resolved @@ -442,8 +460,12 @@ capability is unavailable — emit an 4. SUBMIT follow-up on the SAME thread: tfaRcaTurn(testRunId, message, threadId). turns_used += 1. FAILS ("TFA agent run failed") → resubmit the SAME message on the SAME thread once (per 4b), still counting as a turn. If THAT resubmit also - fails → END (PENDING, note "likely-context-exceeded") immediately, do - NOT continue looping to the turn cap (per 4b/4b-i). + fails (two consecutive same-thread failures) → per 4b, if no restart has + happened yet this run: submit a condensed hypothesis as turn 1 of a + BRAND NEW thread (no threadId), capture the new threadId, turns_used += 1, + go to 2. If a restart already happened once and this (the restarted) + thread also hits two consecutive failures → END (PENDING, note + "likely-context-exceeded") — no second restart. 5. TURN-CAP CHECK: if turns_used >= turnCap and still NEEDS_INFO → END (PENDING, "turn-cap"). else → go to 2 with the new result. 6. EMIT the RCA_OUTPUT block from the captured terminal state.