Agent (FunctionCalling strategy) fabricates tool-call success without executing the tool — correlated with semantic proximity between input data and the Agent's own instruction vocabulary

[abutareqaljaber-code]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.16.1

Cloud or Self Hosted

Cloud

Steps to reproduce

Setup: Agent node, FunctionCalling strategy, model claude-haiku-4-5-20251001.
The Agent has a tool whose parameters accept arbitrary user-supplied data
(e.g. a name or title field passed to a lookup/action tool).

Reproduction:

1. Send a request where the data value shares vocabulary with the Agent's
   own system instructions or tool/workflow terminology (e.g. words like
   "Approval," "Queue," "Confirm," "Deleted," "Submitted" appearing in a
   contact or record name being processed).
2. Observe: on some trials, the Agent skips the actual tool call entirely
   but still returns a fabricated, confident success message.
3. Repeat the identical input multiple times — the failure is
   probabilistic, not deterministic. The same input can pass on one trial
   and fail on the next.

Investigation summary (six rounds of testing; full trial data available
on request):

• Genuinely unrelated/invented words (an invented term plus several
  unrelated real words) showed 0% failure across 15+ trials.
• Words with semantic proximity to the Agent's own rule/workflow/approval
  domain vocabulary showed inconsistent failure rates (roughly 22–67%
  across different words), with individual words flipping between passing
  and failing on repeated trials of the identical input.
• A near-synonym word-substitution attempt (replacing suspect words with
  alternatives) produced no measurable, reliable improvement.
• Ruled out as causes: word rarity, specific blocklisted terms, heading/
  structural formatting, and any fixed vocabulary list. The correlate
  appears to be semantic proximity to the Agent's own instruction-layer
  vocabulary, not a single identifiable trigger.

Impact: currently blocking a delete-confirmation feature in a
production deployment — the affected tools have been permanently disabled
rather than risk a false "deleted successfully" message when nothing was
actually deleted.

Happy to provide full round-by-round trial logs and exact instruction text
on request.

✔️ Expected Behavior

The Agent should either genuinely call the tool, or clearly indicate it
did not/could not, rather than fabricating a plausible success message
when no real tool_response exists in the execution trace.

❌ Actual Behavior

Confident, well-formatted success messages are generated with an empty
tool_responses array in the raw trace — indistinguishable from a genuine
success without inspecting the trace directly. This occurs
probabilistically, correlated with semantic overlap between processed
data and the Agent's own instruction vocabulary, as detailed above.

[dosubot]

Hi @abutareqaljaber-code! I'm Dosu and I'm helping the Dify team.

Your detailed investigation is very helpful. Looking at the FunctionCallAgentRunner code, the root cause becomes clear: Dify's FC agent runner has no validation that a tool was actually invoked before accepting the model's response as a final answer.

Here's how the flow works:

1. The model is called with tools= passed in the request [1].
2. The response is inspected for tool_calls — either in streaming chunks via check_tool_calls() [2] or in blocking results via check_blocking_tool_calls() [3].
3. Only if tool_calls are present does Dify actually invoke ToolEngine.agent_invoke.
4. If the model emits no tool_calls but generates a plausible-sounding text response instead, Dify accepts that text as the final answer — there is no check that a tool invocation was expected but didn't happen.

So what you're seeing is upstream LLM behavior: claude-haiku-4-5, for certain inputs, decides to respond in plain text rather than issuing a tool_call, and fabricates a success message. Dify's FC runner accepts that text and returns it. The empty tool_responses array in the trace confirms no tool was actually dispatched.

Mitigation options within Dify today:

• System prompt enforcement — add explicit instructions like "You MUST call the <tool_name> tool. Never fabricate a result. If you cannot call the tool, say so explicitly." This can reduce the model's tendency to skip the tool call.
• Rename sensitive parameters — if your tool's parameter names echo your instruction vocabulary (e.g. action, status), renaming them to neutral labels (e.g. record_id, target_label) may reduce the confusion.
• Try a different model — larger Claude models may be more reliable about issuing tool calls vs. generating text responses, though this would need testing on your specific use case.
• Switch to ReAct strategy — this uses a different mechanism for tool invocation that may be less prone to silent skipping, though it has its own trade-offs.

A true fix would require Dify to detect when the agent produces a terminal answer with zero tool invocations in a context where a tool call was expected — and either error out or retry. This is not currently implemented in the FC runner [4]. That would be worth filing as a separate feature/improvement request if you'd like to see it prioritized.

Your trial data (22–67% failure rates correlated with vocabulary proximity, 0% for invented terms) is exactly the kind of evidence the maintainers would find useful. Sharing the instruction text and trial logs in this issue would strengthen the case for an application-level mitigation.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[abutareqaljaber-code]

@dosu
Thank you for confirming the root cause with the actual code — this is
exactly the mechanism we suspected but couldn't confirm from our side
without access to the runner internals. Appreciated.

Trial data, as requested, from our six-round investigation:

Round 1: "Test Contact Disambiguation" consistently failed across trials;
a control name ("Priya Sandhu Test Record") consistently passed.

Round 2: Ordinary instruction-adjacent words ("Approval," "Review,"
"Confirm," "Queue") — 2 of 3 failed across trials, ruling out a
"rare/unusual word only" explanation.

Round 3: Unrelated control words ("Marcus," "Priority," "Northgate") —
passed 6/6, supporting vocabulary-overlap (not word choice generally) as
the correlate.

Round 4: 10 of 10 heading/structural instruction words failed on first
trial — ruled out "it's about headings" as the cause.

Round 5: The two words we'd tentatively marked "safe" ("Approval,"
"Queue") both FAILED on a second trial — this was the key finding that
the failure is genuinely probabilistic per-word, not a fixed
pass/fail list.

Round 6: A fully invented word ("Zylquantor") passed 3/3. Further
unrelated real words ("Elephant," "Bicycle") passed 6/6 — stable 0%
failure. Repeat-testing words we'd previously seen fail ("Deleted,"
"Submitted," "Critical," "Actions") showed NONE held a stable rate — two
flipped to majority-pass on re-test. Approximate failure rates across
task-domain-adjacent words ranged 22–67%, never stable, versus a
consistent 0% for genuinely unrelated words across 15+ trials.

We also tried a near-synonym rewrite of the flagged words (e.g.
"Signoff," "Authorize," "Hold" in place of "Approval," "Confirm,"
"Queue") — this showed no statistically meaningful improvement (~22%
failure, within the noise of the original rates), which is consistent
with your finding: the issue isn't specific blocklisted words, it's the
model's response behavior at the point where check_tool_calls()/
check_blocking_tool_calls() come back empty.

One more thing worth trying based on your explanation, which we haven't
tested yet: renaming the tool's own parameter names (not just the data
values or surrounding instruction text, which is what all of the above
covers) — in case parameter names themselves are contributing to the
same vocabulary-overlap effect. We'll test this on our end and report
back if it changes anything.

Given the root cause is now clear at the code level, would it be useful
for us to file that separate feature/improvement request for
terminal-answer validation (detecting a zero-tool-invocation response
when a call was expected, and retrying/erroring instead of accepting it),
or would you recommend we let the maintainers pick this thread up as-is
first?

[dosubot]

Your trial data is excellent — the probabilistic-per-word result from Round 5 and the stable 0% failure rate for unrelated words in Round 6 are the key findings that distinguish this from a simple blocklist effect. The near-synonym rewrite showing no improvement (~22% failure) is strong evidence that the correlate is semantic proximity rather than lexical surface form.

On filing a separate feature request: yes, I'd recommend it. This thread documents the diagnosis thoroughly, but the requested fix — detecting zero-tool-call terminal answers and retrying/erroring instead of silently accepting them — is a distinct runner-layer enhancement. In the FC runner, the loop condition is purely while function_call_state and iteration_step <= max_iteration_steps, with function_call_state defaulting to False at the top of each iteration and only flipping to True if check_tool_calls() or check_blocking_tool_calls() finds tool calls [1] [2] [3]. There is no concept of "expected at least one tool call but got none." Capturing this as a separate improvement request makes it easier for maintainers to scope and prioritize.

For the feature request, a good framing would be: "FC agent runner: detect when a tool-bearing iteration produces zero tool_calls but emits a confident terminal answer, and expose this as a distinct error/retry path rather than silently accepting the fabricated response." The signal is already present — tool_responses is empty and function_call_state stays False [4] — it just isn't acted on.

One related PR worth referencing: #35944 (closed, not merged) added a repeated tool-call guardrail — blocking 3 consecutive identical calls [5]. That's the inverse of your problem (too many calls vs. zero calls), but it shows the runner layer is the right place for this kind of behavioral check and that the pattern has been explored before.

Go ahead and test the parameter-name renaming angle and include those results in the new request — it would help narrow whether the vocabulary-overlap effect operates purely in the prompt text layer or also in the tool schema layer (which gets serialized into the tools= parameter sent to the model).

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[abutareqaljaber-code]

@Docu
Thanks — that framing makes sense. I'll file the feature request as a
separate issue, referencing #40671, and test the parameter-name renaming
angle first so I can include real results rather than just the proposal.
I'll link the new issue here once it's up.