Skip to content

referencing #40671 #40674

Description

@abutareqaljaber-code

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Update, with real test data: tested a parameter-renaming mitigation
(see #40671) using real disposable tasks so the Agent had genuine matches
to find. Results revealed a second, distinct failure mode beyond the
original zero-tool-call fabrication:

Zero-call fabrication Wrong-parameter dispatch
Tool dispatched? No Yes
Runner accepts as success? Yes (no validation) Yes (no validation)
Failure locus Runner loop Model hallucinates argument value
DB side-effect None Row written with wrong identifier
Detection difficulty Empty trace, easy to spot Row looks legitimate, wrong record targeted

The wrong-parameter case is arguably higher severity: a zero-call
fabrication leaves no trace and a human audit catches the absence. A
wrong-argument call writes a row that looks legitimate — an approver
trusting the description rather than independently verifying the actual
identifier would approve based on false context.

Revising the ask accordingly: the proposed guard should validate not
just that a tool was called, but that the tool's actual response contains
a verifiable identifier matching what was requested (e.g. the real
task_id echoed back by the underlying API), not merely that
tool_responses is non-empty. This closes both failure modes with one
mechanism.

Full trial data with real task IDs and the exact "Team not authorized"
evidence for the wrong-parameter case is in #40671.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions