Concurrent tool-using LlmAgents run via `ctx.run_node` crash with "No function call event found for function responses ids" (single_turn appends branchless/scopeless user event to shared session.events)

[leoromero]

🔴 Required Information

Describe the Bug:

When several tool-using LlmAgents are run concurrently as dynamic-workflow nodes (asyncio.gather of ctx.run_node(agent, ...)), runs intermittently crash with:

ValueError: No function call event found for function responses ids: {'tooluse_...'}

raised from google/adk/flows/llm_flows/contents.py _rearrange_events_for_latest_function_response.

Root cause appears to be in run_llm_agent_as_node + the content builder:

1. A bare LlmAgent run via ctx.run_node defaults to mode='single_turn', which force-overrides agent.include_contents = 'none' (workflow/_llm_agent_wrapper.py), regardless of the value set on the agent. This routes content building through _get_current_turn_contents.
2. single_turn then appends the node_input user event to the shared session.events list with no branch and no isolation_scope (_llm_agent_wrapper.prepare_llm_agent_input).
3. A branchless + scopeless event is considered visible to every branch (contents._is_event_belongs_to_branch returns True when event.branch is falsy) and every scope (isolation_scope=None matches the default None).
4. Under concurrency, a freshly-started sibling worker appends its branchless user event into the window between an already in-flight worker's function_call event and its function_response event. On that worker's next LLM step, _get_current_turn_contents walks backward, selects the sibling's user event as the "current turn" boundary, and slices the event list so that it contains the worker's function_response but excludes its own function_call → orphaned function responses → ValueError.

It is a race, so it is intermittent and gets much more likely as the number of concurrent workers and the number of tool rounds per worker increase.

Notably, both of ADK's isolation primitives are broken for this use case:

• use_sub_branch=True → the crash above.
• unique override_isolation_scope per worker (use_sub_branch=False) → no crash, but the agents loop forever (LlmCallsLimitExceededError: Max number of llm calls limit of 500 exceeded) because the scoped contents path never re-presents the tool results to the agent.

The only reliable workaround we found is to run each concurrent tool-using agent in its own session (separate Runner + InMemorySessionService).

Steps to Reproduce:

1. pip install google-adk==2.1.0 litellm==1.85.0
2. Save the script under Minimal Reproduction Code below as repro.py.
3. Run python repro.py → crashes with No function call event found for function responses ids.
4. Run FIX=1 python repro.py → no crash, but infinite tool loop (Max number of llm calls limit of 500 exceeded).
5. Run FIX=2 python repro.py → COMPLETED WITHOUT ERROR (own-session-per-worker workaround).

(No network / API keys needed — the model client is a canned fake that returns parallel tool calls then a final message.)

Expected Behavior:

Running tool-using LlmAgents concurrently as workflow nodes should isolate each agent's conversation so that one worker's node_input/user event cannot be selected as another worker's turn boundary. Each worker should see its own function_call/function_response pair intact and complete normally.

Observed Behavior:

Intermittent crash (scales with concurrency × tool rounds):

File ".../google/adk/flows/llm_flows/contents.py", line 85, in run_async
    llm_request.contents = _get_current_turn_contents(
File ".../google/adk/flows/llm_flows/contents.py", line 709, in _get_current_turn_contents
    return _get_contents(
File ".../google/adk/flows/llm_flows/contents.py", line 637, in _get_contents
    result_events = _rearrange_events_for_latest_function_response(
File ".../google/adk/flows/llm_flows/contents.py", line 224, in _rearrange_events_for_latest_function_response
    raise ValueError(
ValueError: No function call event found for function responses ids: {'tooluse_414dd7_0_1', 'tooluse_414dd7_0_0'}

Environment Details:

• ADK Library Version (pip show google-adk): 2.1.0
• Desktop OS: Windows 11 (also reproduces on Linux)
• Python Version (python -V): 3.14.0 (also reproduces on 3.13/3.12)

Model Information:

• Are you using LiteLLM: Yes
• Which model is being used: Bedrock Anthropic Claude (Sonnet 4.x / Haiku 4.5) in production; the minimal repro uses a canned fake LiteLLM client so no real model is called.

---

🟡 Optional Information

Regression:

Unknown — first observed on 2.1.0; have not bisected against earlier versions.

Logs:

google_adk.google.adk.workflow._node_runner - ERROR - Node execution failed with exception
Traceback (most recent call last):
  ...
  File ".../google/adk/flows/llm_flows/contents.py", line 85, in run_async
    llm_request.contents = _get_current_turn_contents(...)
  File ".../google/adk/flows/llm_flows/contents.py", line 709, in _get_current_turn_contents
    return _get_contents(current_branch, events[i:], ...)
  File ".../google/adk/flows/llm_flows/contents.py", line 637, in _get_contents
    result_events = _rearrange_events_for_latest_function_response(filtered_events)
  File ".../google/adk/flows/llm_flows/contents.py", line 224, in _rearrange_events_for_latest_function_response
    raise ValueError(...)
ValueError: No function call event found for function responses ids: {'tooluse_...'}

Additional Context:

• The damaging interleave does NOT reproduce if all workers fire their tool calls in one synchronized wave (their user events cluster up front, before any function_call). It requires staggered worker starts and/or multiple tool rounds per worker so a sibling's user event lands between a worker's function_call and function_response. The minimal repro forces this with a slow tool, a fast LLM step, staggered starts, and 2 tool rounds.
• run_llm_agent_as_node forcing include_contents='none' means an explicit include_contents='default' on the agent has no effect for node-run agents — this may be surprising independent of the crash.

Minimal Reproduction Code:

"""Reproduces google-adk 2.1.0 'No function call event found for function responses ids'
for concurrent tool-using LlmAgents run as dynamic-workflow nodes. No network / API keys.

  python repro.py          # PROD: use_sub_branch=True   -> ValueError (crash)
  FIX=1 python repro.py     # isolation_scope             -> infinite tool loop (500 llm calls)
  FIX=2 python repro.py     # own session per worker      -> COMPLETED WITHOUT ERROR
"""
from __future__ import annotations
import asyncio, json, os, uuid
from pydantic import BaseModel
from google.genai import types
from google.adk.models.lite_llm import LiteLlm
from google.adk.agents.llm_agent import LlmAgent
from google.adk.workflow import Workflow, node
from google.adk.runners import Runner
from google.adk.sessions.in_memory_session_service import InMemorySessionService
from google.adk.artifacts.in_memory_artifact_service import InMemoryArtifactService
from google.adk.memory.in_memory_memory_service import InMemoryMemoryService
from litellm import ModelResponse
from litellm.types.utils import Message, Choices, ChatCompletionMessageToolCall, Function

N_AGENTS, TOOL_ROUNDS, STAGGER, TOOL_DELAY, LLM_DELAY = 8, 2, 0.05, 0.30, 0.01
MODEL = "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"
FIX = os.getenv("FIX", "")


class Out(BaseModel):
    result: str


def _rounds_done(messages):
    return sum(1 for m in (messages or [])
               if (m.get("role") if isinstance(m, dict) else getattr(m, "role", None)) == "tool")


class FakeClient:
    def __init__(self): self.tag = uuid.uuid4().hex[:6]
    async def acompletion(self, **kwargs):
        await asyncio.sleep(LLM_DELAY)
        r = _rounds_done(kwargs.get("messages"))
        if r >= TOOL_ROUNDS:
            msg = Message(role="assistant", content='{"result":"done"}')
            return ModelResponse(choices=[Choices(finish_reason="stop", index=0, message=msg)], model=MODEL)
        tool_calls = [ChatCompletionMessageToolCall(
            id=f"tooluse_{self.tag}_{r}_{i}", type="function",
            function=Function(name="lookup", arguments=json.dumps({"q": str(i)}))) for i in range(2)]
        msg = Message(role="assistant", content=None, tool_calls=tool_calls)
        return ModelResponse(choices=[Choices(finish_reason="tool_calls", index=0, message=msg)], model=MODEL)


async def lookup(q: str) -> dict:
    await asyncio.sleep(TOOL_DELAY)
    return {"q": q, "rows": 1}


def make_agent():
    model = LiteLlm(model=MODEL)
    model.llm_client = FakeClient()
    return LlmAgent(name="worker", model=model, tools=[lookup],
                    instruction="Use the lookup tool twice.",
                    output_schema=Out, output_key="out", include_contents="default")


async def _own_session(idx):
    runner = Runner(node=make_agent(), session_service=InMemorySessionService(),
                    artifact_service=InMemoryArtifactService(), memory_service=InMemoryMemoryService(),
                    auto_create_session=True)
    msg = types.Content(role="user", parts=[types.Part(text="Source data for this concept.")])
    async for _ in runner.run_async(user_id="u", session_id=f"s{idx}", new_message=msg):
        pass


_FAILURES = []


@node(rerun_on_resume=True)
async def orchestrator(ctx, node_input):
    async def run_one(idx):
        await asyncio.sleep(idx * STAGGER)
        if FIX == "2":
            return await _own_session(idx)
        kw = {"node_input": "Source data for this concept."}
        if FIX == "1":
            kw["use_sub_branch"] = False
            kw["override_isolation_scope"] = f"ds-{idx}-{uuid.uuid4().hex[:6]}"
        else:
            kw["use_sub_branch"] = True
        return await ctx.run_node(make_agent(), **kw)

    results = await asyncio.gather(*[run_one(i) for i in range(N_AGENTS)], return_exceptions=True)
    for r in results:
        if isinstance(r, BaseException):
            _FAILURES.append(repr(r))
    return results


root = Workflow(name="Repro", edges=[("START", orchestrator)])


async def main():
    runner = Runner(node=root, session_service=InMemorySessionService(),
                    artifact_service=InMemoryArtifactService(), memory_service=InMemoryMemoryService(),
                    auto_create_session=True)
    msg = types.Content(role="user", parts=[types.Part(text="start")])
    async for _ in runner.run_async(user_id="u", session_id="s", new_message=msg):
        pass
    print(f"\nFIX={FIX!r}  failures={len(_FAILURES)}/{N_AGENTS}")
    print("COMPLETED WITHOUT ERROR" if not _FAILURES else "FAILED (see traceback above)")


if __name__ == "__main__":
    asyncio.run(main())

How often has this issue occurred?:

• Intermittently (<50%) in production (race; scales with concurrency × tool rounds). The minimal repro above forces the interleave and reproduces it ~consistently (7–8 of 8 workers fail).

[leoromero]

Proposed fix (root-caused to one un-enriched event) + verification

I traced this through the framework and have a one-line-ish fix that resolves the crash on the default use_sub_branch=True path. Summary: the single_turn node-input user event is the only event a node emits that bypasses branch/scope enrichment, which is what makes it globally visible to sibling branches.

Where the bug is

Every event a node emits flows through NodeRunner._enqueue_event → NodeRunner._enrich_event, which stamps event.branch (and event.isolation_scope) from the node's context:

# workflow/_node_runner.py  _enrich_event(...)
if event.branch is None:
    event.branch = ctx._invocation_context.branch
...
if event.isolation_scope is None and ctx.isolation_scope is not None:
    event.isolation_scope = ctx.isolation_scope

But prepare_llm_agent_input (for single_turn) appends the node-input user event directly to ctx.session.events, bypassing that pipeline entirely — so it never gets a branch:

# workflow/_llm_agent_wrapper.py  prepare_llm_agent_input(...)
user_event = Event(author='user', message=agent_input)
...
iso = getattr(ctx, 'isolation_scope', None)
if iso:
    user_event.isolation_scope = iso
ctx.session.events.append(user_event)   # <-- raw append, no branch stamped

It is, in fact, the only raw session.events.append in the entire workflow path. A branch-less event is treated as visible to every branch (contents._is_event_belongs_to_branch returns True when event.branch is falsy), so under concurrency a sibling's input event can be selected as another worker's "current turn" boundary, slicing out that worker's own function_call.

Why stamping the branch is correct (not a workaround)

The framework deliberately treats a branch-less user event as globally visible — that's the intended behaviour for the root user turn that every sub-agent should see. See the branch filter that explicitly special-cases it:

# agents/invocation_context.py
if current_branch:
    ... event.branch == self.branch
    or (event.branch is None and event.author == "user")   # root user input is shared

The bug is that prepare_llm_agent_input reuses that same "branch-less user event" shape for a per-worker private input, so it wrongly inherits root-input semantics. Stamping the worker's branch restores the intended distinction between "the shared root user turn" and "this node's private input."

The fix

# workflow/_llm_agent_wrapper.py  prepare_llm_agent_input(...)
    iso = getattr(ctx, 'isolation_scope', None)
    if iso:
        user_event.isolation_scope = iso
    # Scope this input event to the node's branch so it is not globally visible
    # to concurrent sibling nodes (mirrors NodeRunner._enrich_event). Without this,
    # a sibling's branch-less user event can be picked as this worker's turn
    # boundary and slice out its own function_call.
    branch = getattr(ctx._invocation_context, 'branch', None)
    if branch:
        user_event.branch = branch
    ctx.session.events.append(user_event)

Under use_sub_branch=True, ctx._invocation_context.branch is already the unique sub-branch set by _node_runner._create_child_ctx. The worker's function_call/function_response events already carry that branch via _enrich_event; this just makes the input event consistent with them, so _get_current_turn_contents only ever sees the worker's own input as the turn boundary.

Notes:

• It only adds a branch to an event that should always have had one — no behaviour change for non-concurrent runs (an agent always sees its own branch, and still sees the shared root user turn).
• If you'd prefer no duplication of the two lines from _enrich_event, I'm happy to extract a small shared helper, e.g. apply_ctx_scope(event, branch, isolation_scope), called from both sites. I went with the localized stamp here to keep the blast radius minimal (cold path, no edit to the hot _enrich_event).

Verification

Using the staggered-interleave reproducer from the issue (which fails 7–8 / 8 on stock 2.1.0 with use_sub_branch=True), monkeypatching prepare_llm_agent_input to add exactly the block above:

FIX=branch-stamp  use_sub_branch=True  failures=0/8
COMPLETED WITHOUT ERROR

Repeatable green across runs. The original crash and the FIX=2 own-session workaround behave exactly as before; only the default path is now correct.

Scope of this fix

This resolves the crash (use_sub_branch=True). The separately-noted override_isolation_scope infinite-loop symptom (FIX=1: the scoped contents path never re-presents tool results, so the agent loops to the 500-call cap) is a different defect in the scope-based contents path and isn't addressed here — happy to open a separate issue for it if useful.