Skip to content

图片验证模块、拖动验证模块识别失败;用户可以主动申请中断浏览器控制状态而不中止任务,处理后交接恢复 | Fix Image and Slider Verification Failures and Allow User-Initiated Browser Takeover Without Terminating the Task #56

Description

@Ljy-0827

问题描述

当前图片验证与拖动验证模块的处理行为不稳定,主要表现为以下两类情况:

  1. 部分验证场景中,Agent 未请求人工介入,但验证流程仍然顺利完成;
  2. 部分场景中,Agent 无法自行完成验证,却也没有主动请求用户协助,导致任务停滞或反复尝试。

目前尚不确定上述问题是由 Observation 未能准确呈现验证模块的页面状态和交互信息导致,还是由于 Agent 未能正确识别验证步骤、判断自身能力边界或触发人工协助流程。需要结合运行轨迹进一步排查 Observation 与 Agent 决策过程。

预期行为

当 Agent 无法继续完成图片验证、拖动验证,或系统观察到 Agent 长时间停滞、重复尝试、执行结果持续不符合预期时,应允许人工主动介入处理但不终止整个任务的运行,和普通接管浏览器的状态一致。

Problem Description

The current handling of image-based and slider verification modules is inconsistent, primarily in the following two situations:

  1. In some verification scenarios, the Agent completes the verification successfully without requesting human intervention.
  2. In other scenarios, the Agent is unable to complete the verification independently but also fails to request user assistance, causing the task to stall or enter repeated attempts.

It is currently unclear whether these issues are caused by the Observation failing to accurately represent the state and interaction information of the verification module, or by the Agent failing to correctly identify the verification step, recognize the limits of its capabilities, or trigger the human-assistance flow. Further investigation of the execution traces is required to distinguish between Observation-related issues and Agent reasoning or decision-making issues.

Expected Behavior

When the Agent is unable to complete an image-based or slider verification, or when the system detects that the Agent has remained stuck for an extended period, repeatedly attempted the same action, or continuously produced outcomes that do not match expectations, the user should be allowed to proactively intervene without terminating the overall task.

The system should enter the same state as a standard browser takeover, preserving the current task execution and allowing the Agent to resume after the user has completed the required operation and handed control back.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions