feat: robust support for remote job re-attachment

[ruoyuwang1995nya]

This pull request introduces robust support for remote job handoff and recovery in the step execution flow, ensuring that long-running jobs are not interrupted or duplicated if an executor times out or crashes. The changes allow step executors to gracefully detach from active remote jobs, mark their status as waiting, and later re-attach to collect results, preventing wasted compute and duplicate submissions. The recovery logic is unified and explicit, improving reliability and maintainability.

Remote job handoff and executor timeout handling:

• Added a mechanism for step executors to enter a waiting state (rather than failing) if their wall-clock timeout expires but a tracked remote job is still running, with a bounded grace period before handoff. The execution graph node is updated directly to reflect this state, ensuring deterministic handoff and enabling later re-attachment. [1] [2] [3] [4] [5] [6]

• Introduced the _await_step_completion helper to manage timeouts and grace windows without cancelling the in-flight executor, and to distinguish between true timeouts and remote job handoffs.

Remote job recovery and re-attachment:

• Refactored recovery logic to track in-progress remote jobs, using a new _REMOTE_JOB_IN_PROGRESS_STATUSES set, and to allow nodes in waiting state to resume execution and collect results once jobs settle. [1] [2] [3] [4] [5]

• Added explicit instructions for re-attaching to existing remote jobs in the executor's prior_context, preventing duplicate submissions and ensuring idempotency. [1] [2]

Interface and status updates:

• Extended the allowed NodeStatus values to include waiting, reflecting the new handoff state in the execution graph and agent logic.

Refactoring and code hygiene:

• Updated imports and helper functions to support the new handoff and recovery mechanisms. [1] [2]