Conversation
…ith a MESSAGE prefix match
eligotts
changed the title
| prev_turn_completion_ids = prev_turn_tokens["completion_ids"] | ||
| prev_turn_ids = prev_turn_prompt_ids + prev_turn_completion_ids | ||
|
|
||
| def normalize_for_comparison(value: Any) -> Any: |
There was a problem hiding this comment.
should we make this a general message_util? seems useful in other places too? also, vaguely remember we have a similar util to this alr but might be wrong
|
|
||
| return 0 | ||
|
|
||
| # we add suffix_ids to prev_turn_ids. suffix_ids are tokens that are added |
There was a problem hiding this comment.
i know unrelated to this pr but can i think we might be able to remove the suffix part since we tokenize the env_response_ids = full_ids[len(prev_turn_ids) :] and not tokenize env response ids in isolation
There was a problem hiding this comment.
yep looks like some circular logic, might be kinda confusing because env_response_ids now might contain suffix/delimiters of assistant message but functionally will be the same
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| best_step_tokens = step_tokens | ||
| if best_prefix_len == len(normalized_prompt_messages): | ||
| break | ||
|
|
There was a problem hiding this comment.
Per-turn backward scan may be expensive
Medium Severity
get_prompt_ids() now walks backward over the entire state["trajectory"] and calls to_native_prompt() per step until it finds the best prefix, which can add significant overhead on long trajectories and slow every generation turn.
| if best_step_tokens is None: | ||
| return None | ||
| return best_step_tokens["prompt_ids"] + best_step_tokens["completion_ids"] | ||
|
|
There was a problem hiding this comment.
Prefix match can miss equivalent messages
Medium Severity
get_prompt_ids’s new message-level prefix matcher compares normalized message objects for strict equality, which can differ across representations (e.g., to_native_prompt emitting {"content": None} while incoming prompt_messages omits content, or other default/extra fields). This can produce false “no prefix match”, disabling the token route unexpectedly.
There was a problem hiding this comment.
this is not valid right, because we expect the trajectory step list to be append only essentially right? so even if there is some anomalies in the message, it would be the same across all steps
There was a problem hiding this comment.
yeah i don't think we need to worry about this. the flow is native response (OAI types) -> custom types (ToolCall, AssistantMessage, etc) which get stored in TrajectoryStep. as long as get_prompt_messages doesn't too anything too crazy as is extending from some existing TrajectoryStep (like for RLMs, we have the individual subagent trajectories. but for true compaction, we dont do this extension, so we wouldn't find any message prefix), those prompt messages go from custom types -> native prompt (OAI types), then in this code snippet we are comparing to previous trajectory steps which also get transformed to native types. so everything is going from custom types -> native types through to_native_response so we shouldn't really be missing anything.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
|
|
||
| prev_turn_ids = await find_largest_prefix_match_tokens() | ||
| if prev_turn_ids is None: | ||
| return None |
There was a problem hiding this comment.
Prefix match ignores tool-dependent tokenization
Medium Severity
find_largest_prefix_match_tokens() selects a trajectory step using only a message-level prefix comparison, but the stitched prev_turn_ids are later combined with full_ids produced by tokenize(..., tools=oai_tools). If the effective tool set differs from when the matched step’s tokens were produced, prev_turn_ids may not align with full_ids, yielding incorrect env_response_ids and an invalid prompt for /chat/completions/tokens.
2 participants
Description
Best effort tito. instead of assuming extension and looking back at strictly last trajectory step, we walk backward until we find a MESSAGES level prefix hit
tested on both wiki-search and eligottlieb/poker-multiagent, which was giving the loud failure previously with TITO as it does explicit rewriting of history (like context folding)
wiki search no regression:
tests on true branching envs (multi-agent setups):
what this shows is for each agent trajectory (extension per agent turn, but because all stuffed into same trajectory we have 'branching'), we are successfully retrieving the latest trajectory step and doing token in from that point.
Type of Change
Testing
uv run pytestlocally.Checklist
Additional Notes
Note
Medium Risk
Changes how token-stitching selects prior-turn tokens and adds a new fallback path; mistakes could cause silent behavior changes (more MITO calls) or incorrect token prefixes in production.
Overview
Makes token-stitching (TITO) best-effort:
get_prompt_idsnow scans the trajectory backwards to find the largest message-level prefix match (normalizing message structures) instead of assuming the last step matches, and returnsNonewhen no match exists.Updates
get_native_responseto fall back to the standard/chat/completionspath whenget_prompt_idsreturnsNone, avoiding broken/chat/completions/tokensrequests on history rewrites/context folding. Adds async tests covering largest-prefix selection, no-prefix behavior, and ensuring the correct route is used depending on whether prompt token IDs are available.Written by Cursor Bugbot for commit 6ae6aff. This will update automatically on new commits. Configure here.