fix(agents): fail a generation whose answer is a tool call written as text

[arantespp]

Closes #869.

A model that writes its tool invocation out as assistant text instead of making one produced a generation indistinguishable from a real answer — finishReason: "stop", one step, no tool-call part, status: completed, error: null — with the blob as output.content and the tool never run:

```json
{"name": "get-fundamental-truth", "arguments": {}}
```

Reported on grok-4.5 / xai.responses at roughly 1 in 10 runs. It is a silent data-integrity failure: on a board the card advanced, the value was approved by the next column, published, indexed, and handed to the next agent — which had nothing to work from and invented an unrelated subject. Every column reported success.

The structured path already handles the same misbehavior correctly: an output_schema violation fails loudly with OUTPUT_SCHEMA_VALIDATION_FAILED (#867). This closes the schema-less path the same way.

What changed

• textEncodedToolCall.ts — findTextEncodedToolCall reports a final assistant text that is entirely a tool-call blob. Deliberately narrow, because a false positive fails a generation that was fine. All three must hold: the text (after a wrapping markdown fence is stripped) is exactly one JSON object or an array of them; every key is tool-call vocabulary (name/tool/tool_name/function, arguments/args/parameters/input, id, type); and the name is a tool bound to that agent. Prose around the JSON, one key outside the vocabulary, an unbound tool name, or an agent with no tools — all left alone.
• All three completion paths consult it:
  • the initial turn, in buildCompletedGenerationResult — after the trace is written, so the offending text survives for post-mortem, and before the record is marked completed;
  • the tool-outputs continuation, via a new recordContinuationFailure in generationLifecycle (that path has no try/catch above it, so a bare throw would strand the generation in requires_action with nothing recorded);
  • a stream's onEnd — the text has already gone down the wire and cannot be recalled, but the generation and its trace are still recorded failed.
• New error code TEXT_ENCODED_TOOL_CALL (502), upstream-caused like AI_PROVIDER_ERROR, with meta.tool_name naming the tool alongside the usual generation_id / trace_id.
• Agents with an output_schema are exempt — that path validates the model output itself and already fails loudly, and its content is the serialized object.
• Docs: a "A tool call written out as text" section in the Agents module page, and the 502 descriptions on both generation endpoints in agents.yaml.

content resolution is unchanged, deliberately: ai@7's result.text is steps.at(-1).text, so it already comes from the final step and never from an earlier one. finalStepText spells the same rule out for the stream path, which only receives the step array. The issue also suggested dropping a fallback to toolName — there is no such fallback anywhere in src, so there was nothing to remove.

Tests

Red/green throughout — the entry-point tests were confirmed failing against the unfixed path before the guard was wired in (3 detection tests red, both "leave it alone" tests green).

• tests/unit/tests/lib/textEncodedToolCall.test.ts — the detector directly, over every improvised shape and every ordinary answer that must survive untouched (keep-list: pure function, large input space).
• tests/unit/tests/rest/agentGeneration.test.ts — the wiring at the entry point, against the local OpenAI-compatible stub: the blob fails with 502 and the generation is recorded failed; the trace keeps the offending step; an ordinary answer from the same tool-bound agent still completes; an agent with no tools is left alone; an output_schema agent is left to the schema validator; the tool-outputs continuation fails on it too.

pnpm typecheck, pnpm eslint --fix, and docs-lint pass. Full server suite: 176 suites, 4895 tests, all passing. New module coverage: 100% statements / 96% branches.

Smoke tests were not extended: reproducing this needs a provider that misbehaves on demand, which the Ollama stack cannot do without adding a new capability to ollamaToolChoiceProxy.mjs. They were also not runnable here (no Docker daemon in this environment).

Open questions

Q: Should the bare-tool-name variant (content == "get-fundamental-truth", no JSON
   wrapper) also fail the generation?
A: No — resolved by long-term. Rejecting a text that is exactly a bound tool name
   would break router agents whose legitimate job is to output a tool name, and the
   reported instance of that variant had the real answer in a later step, which is a
   content-resolution question rather than a detection one. Checked this session:
   ai@7 `result.text` is `steps.at(-1).text` (verified in the installed dist), so
   content already comes from the final step.

Q: Fail a streaming generation, given its text has already been delivered?
A: Yes, record it — resolved by pareto. The stream body cannot be recalled, but
   leaving the record as `completed` makes the failure invisible; recording `failed`
   on the generation and trace worsens nothing (checked: streaming rejects
   `output_schema` upfront, and every other `onEnd` write is already fire-and-forget).

Q: Detect at the adapter level, as the issue suggests, or at the completion paths?
A: Completion paths — resolved by long-term. The check needs the agent's bound tool
   names and its `output_schema`, neither of which the provider adapter has; placing
   it at the three points that build a completed result covers every provider at once,
   including future ones.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mug6x5hctSkTRP6SPktU5q

---

Generated by Claude Code

[github-actions]

Deploy Outputs

Package	Stack	Output Key	Output Value
@soat/website	SoatWebsite-claude-issue-status-check-v40rv5	BucketWebsiteURL	http://soatwebsite-claude-issue-status-check-staticbucket-bldvjjagjurk.s3-website-us-east-1.amazonaws.com