FC agent runner: detect zero-tool-call terminal answers when a tool call was expected, instead of silently accepting a fabricated response

[abutareqaljaber-code]

Self Checks

• I have read the Contributing Guide and Language Policy.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Follow-up to #40671, where the root cause of a tool-call fabrication bug
was diagnosed down to the runner logic, confirmed by @dosu against the
actual fc_agent_runner.py source.

In the FC agent runner's main loop, function_call_state defaults to False
at the top of each iteration and only flips to True if check_tool_calls()
or check_blocking_tool_calls() detects a tool call in the model's
response. If the model returns plain text instead, the loop exits after
one iteration, and that text becomes final_answer with no validation that
a tool call was actually expected or required.

The signal that this happened is already present — tool_responses stays
empty and function_call_state stays False — it just isn't acted on.
There's currently no way to distinguish "the agent had nothing to do"
from "the agent was supposed to call a tool and didn't."

We hit this in production: an agent configured to delete records via a
tool call sometimes fabricated a "deleted successfully" message with zero
actual tool invocation in the trace. We've had to permanently disable the
affected tools rather than risk this — which is exactly the kind of gap
this request is meant to close.

What I'd like to see: a way to mark specific tools (or agent
configurations generally) as requiring a tool call when invoked in a
context that expects one, and detect when a terminal answer is produced
with zero tool invocations in that context. When detected, expose this as
a distinct error or retry path, rather than silently returning the
model's fabricated text as a successful answer. A structural fix at the
runner level would let us (and presumably others) re-enable delete/write-
action tools with confidence, rather than relying entirely on the
underlying model's tool-calling reliability.

2. Additional context or comments

Alternatives already tried, none of which address the underlying gap:

• System prompt enforcement ("You MUST call the tool") — reduces but does
  not eliminate the failure, since it's a model-level probabilistic
  behavior, not something promptable away.
• Renaming tool parameters to avoid vocabulary overlap — currently
  testing this separately; will share results in Agent (FunctionCalling strategy) fabricates tool-call success without executing the tool — correlated with semantic proximity between input data and the Agent's own instruction vocabulary #40671 if useful.
• Switching agent strategy to ReAct — a different mechanism, but a real
  behavioral trade-off, not evaluated yet for our use case.

Full diagnosis and six rounds of trial data (showing the failure is
probabilistic and correlated with semantic overlap between tool-adjacent
vocabulary and processed data, not fixable via prompting alone) are in
#40671.

3. Can you help us with this feature?

• I am interested in contributing to this feature.

[dosubot]

Thanks for filing this — the framing is precise and the code path confirms it.

Looking at fc_agent_runner.py, here's the exact gap:

1. function_call_state is reset to False at the top of every iteration [1].
2. It only flips to True if check_tool_calls() (streaming) or check_blocking_tool_calls() (blocking) finds tool calls in the response [2] [3].
3. If neither fires, the loop exits and whatever text the model produced becomes final_answer — with no check that tools were available and a call was expected [4].
4. The tool_responses list stays empty and is never inspected post-loop [5].

The only existing guard is AgentMaxIterationError, which fires when the model keeps calling tools beyond the limit [6] — the inverse of your problem.

What a fix would need to do at the runner level:

• Detect that prompt_messages_tools was non-empty going into an iteration (i.e., tools were offered to the model) but tool_calls came back empty and response is non-empty plain text.
• On first occurrence: either raise a typed error (e.g. AgentZeroToolCallError) that surfaces in traces, or trigger a configurable retry with a stronger nudge prompt before accepting the text answer.
• Expose a flag in agent configuration — something like require_tool_call: bool or tool_call_enforcement: strict | best_effort — so this is opt-in rather than a breaking change for agents where a plain-text terminal answer is intentional.

The signal you identified — tool_responses empty + function_call_state False + prompt_messages_tools non-empty — is sufficient to detect the condition without any model-side changes.

A few design questions worth raising for maintainers to scope this:

• Retry count: should the runner retry N times before erroring, or error immediately? A single retry with tool_choice="required" (where supported by the model's API) might be the simplest path for write/delete-action tools.
• Scope: per-tool flag, per-agent-config flag, or both? Per-agent is simplest to implement; per-tool is more granular.
• Error surface: should this be a hard error visible to the end user, or an internal log + silent retry? For destructive tools (delete, write), a hard error seems safer.

This is a well-scoped request with a clear code location. The reference to #40671 for the diagnostic data keeps this one focused on the fix.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[abutareqaljaber-code]

@dosu
Separate question from the bug/feature threads above: is there any
supported, documented way for a third-party AI assistant to get direct,
programmatic access to a Dify workspace's configuration — specifically,
reading/toggling an Agent's Toolbox tool states, editing tool parameter
names, or reading Agent execution traces — without going through the
chat/UI directly?

I'm aware of the MCP Server feature for exposing a published app as a
conversational endpoint, but that's not what I mean — I'm asking about
administrative/configuration-level access, not conversational access.

If this exists, what's the actual mechanism (an admin API, a specific
endpoint, an OAuth scope, something else)? If it doesn't exist today, is
it something on Dify's roadmap, or a known limitation?

[dosubot]

The short answer: there is no supported, documented, stable admin/configuration API intended for third-party programmatic access today. Here's what actually exists and what the constraints are:

What exists internally (console API)

Dify has a set of internal console API routes under /console/api/agent that cover exactly what you're asking about — reading agent configuration, managing tool states, reading logs/traces, etc. [1] The relevant endpoints include:

Method	Path	Purpose
GET/PUT/DELETE	/agent/<agent_id>	Agent config detail
GET/PUT/DELETE	/agent/<agent_id>/build-draft	Draft config (where tool parameters live)
GET	/agent/<agent_id>/logs	Conversation/execution logs
GET	/agent/<agent_id>/statistics/summary	Usage stats

However, all of these routes require setup_required, login_required, and account_initialization_required authentication, and editing operations additionally require edit_permission_required [1]. They use session-cookie-based authentication tied to a logged-in console user — not an API key or OAuth token. There is no supported mechanism to authenticate to these endpoints as a third-party service.

What the public service API covers

The service API (accessed with app-scoped API keys at /v1/...) is designed for runtime use of a published app — sending messages, retrieving conversation history. It does not expose configuration, tool states, or execution trace internals.

The gap

Issue #38659 was filed by another user asking for exactly this: a first-party GET /v1/apps/discovery-style endpoint exposing agent configs, tool provider details, and trace correlation data — because integrators currently have to hit the PostgreSQL database directly, which is fragile and insecure. That issue is open with no resolution as of now.

Practical options today (self-hosted only)

If you're self-hosting and need this for tooling/observability purposes, your options are:

1. Console API with a bot account session — technically possible but unsupported, fragile, and breaks with session expiry.
2. Direct DB access — what other integrators have resorted to [2]; couples you to the internal schema.
3. Tracing export — for execution traces specifically, Dify supports exporting trace data to external backends (e.g. via LangSmith, LangFuse integrations), which gives you trace data without touching the API layer. (Note: I couldn't fully verify the OTEL-specific export path, so treat this as a direction to investigate rather than a confirmed feature.)

There's no OAuth scope system or admin API key mechanism currently. For cloud Dify, none of the DB/session workarounds are available at all.

If programmatic config access is important to your use case, adding a 👍 or comment to #38659 would help signal demand to the maintainers — that request is the most directly relevant prior art.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[bobleer]

The guard needs a caller-owned expectation, not merely “tools were available”; plenty of valid agent turns choose no tool. The run request should declare a required tool/effect contract, and the runner should require a matching call plus result receipt. For destructive tools, terminal text without that receipt should settle as failed/unknown—not success, and not a blind retry that might duplicate a mutation.

[abutareqaljaber-code]

@dosu
Update: tested the parameter-renaming mitigation suggested above
(status_code → record_state on request_delete_clickup_task), with a
proper test this time — real disposable ClickUp tasks created first, so
the Agent had genuine matches to find, not an empty list.

Result: the rename did not fix the underlying issue, and testing it
surfaced a second, related failure mode worth documenting separately.

3 trials, same task-naming pattern as the original investigation
("Approval Rename Trial 1", "Queue Rename Trial 2", "Confirm Rename
Trial 3"):

• Trial 1: correct — real tool call, correct task_id, real approval_queue
  row created.
• Trial 2: a REAL tool call was made and a real approval_queue row was
  created, but with a task_id that does not match the actual task —
  querying that task_id directly returns "Team not authorized," meaning
  it's either invalid or belongs to an inaccessible workspace. This is
  distinct from the original bug: not a fabricated success message, but
  a fabricated/incorrect parameter on a genuine tool call.
• Trial 3: the original bug, unchanged — confident "sent for sign-off"
  chat message, zero real tool call, nothing in the database.

So: 1 of 3 clean, 1 of 3 the original zero-call fabrication, 1 of 3 a new
failure mode (real call, wrong data). The parameter rename does not
appear to change the underlying failure rate or mechanism.

Flagging Trial 2 in particular — if a human approver isn't independently
verifying the task_id against the real task (rather than just trusting
the description shown), this failure mode could result in approving a
deletion request that either fails silently or, in a differently
structured system, could target an unintended real record.

[dosubot]

Thanks for running this properly with real tasks — the Trial 2 result is the most important finding here and deserves to be called out clearly.

Trial 2 and Trial 3 are distinct failure classes. In Trial 3, the model emitted no tool_calls at all — function_call_state stayed False, the tool_calls list remained empty, and the tool dispatch loop was never entered. The runner accepted the model's fabricated text as a final answer because it has no zero-call guard [1]. In Trial 2, the model did emit a tool call, ToolEngine.agent_invoke was invoked, and a real row was written — but the model hallucinated the task_id argument. That's a parameter-level hallucination on a genuine dispatch, not a dispatch-level fabrication.

	Trial 3	Trial 2
Tool dispatched?	No	Yes
Runner accepted as success?	Yes (no validation)	Yes (no validation)
Failure locus	Runner loop (missing zero-call guard)	Model (hallucinated argument value)
DB side-effect	None	Row written with wrong foreign key
Detection difficulty	Empty trace, no tool_response	Row exists, looks valid, wrong record targeted

Trial 2 is arguably higher severity for your use case. A zero-call fabrication at least leaves no footprint — your approval_queue table is clean and a human audit can catch the absence. A wrong-task_id call writes a row that looks legitimate but targets a non-existent or inaccessible record, and an approver who trusts the description rather than independently verifying the task_id would approve based on false context.

@bobleer's point above is directly relevant: a guard requiring a result receipt from a specific named tool — not just "a tool was called" — would catch Trial 2 as well as Trial 3, if the receipt validation includes checking that the tool_response contains a verifiable identifier (e.g. the actual task_id echoed back by the ClickUp API). A blind retry would be dangerous precisely in this scenario — retrying a wrong-task_id call could write another bad row.

The parameter rename is confirmed not to help. Your data now covers three failure modes from 3 trials: 1 clean, 1 zero-call fabrication, 1 wrong-parameter-on-genuine-call. I'd suggest documenting Trial 2 explicitly in the feature request body — the wrong-param case strengthens the argument for result-receipt validation considerably, since it shows the failure surface extends beyond "was a tool called at all" into "were the arguments correct."

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[yun520-1]

@bobleer — fully agree the guard needs a caller-owned expectation, and I'd push it one step further: the expectation shouldn't be "tools were available" or even "the caller declared a tool expectation" — it should be derived from what the task type implies. We build a pure-rule discrimination layer (no LLM), and this exact class of bug — "state was supposed to change but nothing happened, and the agent reported success anyway" — is why our gate separates scope from verification.

Concretely: classify the incoming task before the run, not during. Three buckets:

1. Read-only / pure-answer — zero-tool terminal answers are legitimate. No guard.
2. Side-effect task (delete, write, send, mutate) — the run must show a tool invocation with a success receipt before a terminal answer is accepted. Zero-tool terminal = fabricated, regardless of what the text claims.
3. Ambiguous — the runner can't tell. Default to the stricter bucket (treat as side-effect) and log the classification so false positives are visible and adjustable.

The key property this gives you is that the check runs on task class, not on tool availability — so "plenty of valid agent turns choose no tool" stays true for bucket 1, while bucket 2 can never silently pass. That's the same split we use between scope-check (does this request even belong here) and verifier-grant (did the claimed effect actually happen). The fabrication you hit in production ("deleted successfully" with zero invocation) is bucket 2 answering with a bucket 1 shape — the missing step is exactly the classification.

One implementation note: the signal you already identified (tool_responses empty + function_call_state False) is the right trigger, but it only fires post-hoc. If you classify at the start, you can also fail loud at the boundary — surface "expected tool call, got terminal text" as a distinct retryable error instead of a silent success, which is what lets you re-enable those delete/write tools with confidence.