process/revalidate batches fail with zero retries when the agent output does not parse (parse step is outside the retry loop)

[teslerbar]

Summary

The agent SDK plugins retry on transient errors (MAX_ATTEMPTS = 3 + backoff), but the output-parsing step runs after and outside that loop. So when the agent completes its investigation successfully and then returns text that doesn't parse into the expected schema (malformed/partial JSON, prose around the JSON, wrong shape), the batch fails immediately with no retry, and every file in it is marked status=error.

Because the agent finished, this isn't a transient failure — but it's often nondeterministic: a re-run of the same files usually parses cleanly. The current code never gives it that second chance within the run; the files only get re-analyzed on the next full process/revalidate invocation.

Where

processor/src/agents/claude-agent-sdk.ts (and the same shape in processor/src/agents/codex-sdk.ts). Simplified from the 2.0.12 build:

for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  // ... run the agent ...
  if (resultText) break;                                     // got a result → stop retrying
  if (attempt >= MAX_ATTEMPTS || !isTransientError(lastError)) break;
  await backoff(attempt);
}
// ↓↓↓ parsing happens OUTSIDE the retry loop ↓↓↓
let results;
try {
  results = parseInvestigateResults(resultText, batch);     // throws on malformed output
} catch (err) {
  writeParseFailureDebug({ projectId, phase: "investigate", resultText, error: err, batch });
  throw err;                                                 // ← no retry, batch fails
}

This affects 4 sites: parseInvestigateResults (investigate) and parseRevalidateVerdicts (revalidate), in both the claude-agent-sdk and codex-sdk plugins.

Impact

On a 13-batch process run we saw 4 batch(es) failed purely from unparseable output, leaving ~13 files at status=error for that run. Across a daily multi-repo scan this compounds — files silently drop out of coverage until the next run happens to parse them. (#82's debug dump helps post-mortem, but the batch still fails.)

Proposed fix

Treat a parse failure as a retryable condition by moving the parse inside the attempt loop, ideally with a corrective re-prompt on the next attempt:

for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  // ... run the agent (attempt > 1 appends a corrective instruction) ...
  if (resultText) {
    try {
      results = parseInvestigateResults(resultText, batch);
      break;                                                  // parsed → done
    } catch (err) {
      lastError = `output did not parse: ${err.message}`;
      writeParseFailureDebug({ ...,  attempt });
      if (attempt >= MAX_ATTEMPTS) throw err;
      reprompt = "Your previous reply did not parse as the required JSON schema. "
               + "Reply with ONLY the JSON object, no prose or code fences.";
      continue;                                               // retry with correction
    }
  }
  if (attempt >= MAX_ATTEMPTS || !isTransientError(lastError)) break;
  await backoff(attempt);
}

Minimal version (no re-prompt): just wrap the parse in the loop and continue on failure so the existing MAX_ATTEMPTS budget covers parse failures too. A schema-guided re-prompt would lift the success rate further.

Related: #33 (transient rate-limit retries) covers the other half of the retry story; this is the parse-failure half.

Happy to open a PR if that'd help.