Treat a step-executor timeout with an active remote job as a waiting handoff

[Copilot]

A step executor has a fixed 3600s wall-clock timeout, but an e2b job it submits is durable and tracked independently by RemoteJobStore + RemoteJobMonitor. Previously any timeout returned needs_replanning, which the orchestrator turned into a terminal failed node with blocked dependents — even while the job was still running — and re-attachment then only worked by luck, if the replacement plan happened to reuse the same node id and template so the submission idempotency key matched.

The waiting state and _active_remote_job lookup that solve this already existed, but only ran on process-restart reconciliation. This routes the live-timeout path into that same seam rather than extending or removing the timeout (which would make a bounded LLM session the single point of failure for an arbitrarily long job).

Timeout path (step_executor_runner.py)

• _await_step_completion replaces asyncio.wait_for, so a timeout no longer unconditionally kills the executor.
• On timeout, checks the job store for a job still owned by the node. If found, grants one bounded grace window (STEP_REMOTE_JOB_GRACE_TIMEOUT, default 300s) for nearly-finished work, then releases the executor and returns status: "waiting" instead of needs_replanning.
• _mark_node_waiting_on_remote_job writes the status and job identity directly onto the graph node, so the handoff doesn't depend on the orchestrator LLM calling set_node_status.

Deterministic re-attachment

• recovery.py: new active_remote_job_for_attempt() reloads the persisted job reference (record_remote_job_reference writes through the durable record, so a live attempt only learns of it on its next heartbeat). reconcile_recovery_state now handles waiting attempts — node stays waiting while the job is in progress, returns to pending once it settles so a fresh executor collects results. A planner-initiated resume is explicitly not undone; without that guard the retry would deadlock.
• _remote_job_prior_context injects the job identity and explicit re-attach instructions into the retry executor's context, replacing reliance on an accidental idempotency-key match.
• get_ready_nodes surfaces remote_job on ready nodes and reports waiting_nodes; orchestrator, planner, and step-executor instructions updated to treat waiting as a handoff, not a failure.

Flash mode

run_flash_step's fallback node id was flash_{counter} with an incrementing counter, so a retry got a fresh id, broke the idempotency key, and could submit a duplicate sandbox. Now derived from the action:

if label:
    node_id = label.lower().replace(" ", "_")[:40]
else:
    node_id = f"flash_{hashlib.sha256(action.encode()).hexdigest()[:12]}"

Notes

• SUB_STEP_TIMEOUT is unchanged; the grace window is deliberately secondary.
• succeeded/collecting are excluded from the "still in progress" set used to decide waiting-vs-pending — those mean results are ready, which is exactly when the node should run again.
• waiting added to graph_logger's node status literal; docs/remote_job_monitoring.md updated.