fix: improve eval resume flow consistency and add validation

[Chibionos]

Problem

The eval flow had two issues:

1. Inconsistency in handling resume options compared to the debug flow:

   • Debug flow (cli_debug.py): Always explicitly passes UiPathExecuteOptions(resume=resume) regardless of the resume value
   • Eval flow (_runtime.py): Only passed UiPathExecuteOptions(resume=True) when resume=True, but passed no options at all when resume=False

2. Missing validation for the unsupported scenario of running multiple evaluations with resume mode enabled

Code Comparison (Issue #1)

Before (eval flow):

if self.context.resume:
    options = UiPathExecuteOptions(resume=True)
    result = await execution_runtime.execute(input=..., options=options)
else:
    result = await execution_runtime.execute(input=...)  # ❌ No options!

Debug flow (correct pattern):

result = await debug_runtime.execute(
    ctx.get_input(),
    options=UiPathExecuteOptions(resume=resume),  # ✅ Always explicit
)

Issue #2: Missing Validation

Resume mode relies on checkpoint discovery using a consistent thread_id. When multiple evaluations run in parallel, each creates separate runtime contexts, making it impossible to determine which checkpoint to resume from. This was not being caught, potentially leading to confusing runtime behavior.

Solutions

1. Consistent Options Passing

Made the eval flow consistent with the debug flow by:

• Always passing UiPathExecuteOptions explicitly, regardless of resume value
• Simplifying the if/else logic by separating input determination from execution
• Adding a comment explaining the consistency rationale

After:

if self.context.resume:
    logger.info(f"Resuming evaluation {eval_item.id}")
    input = input_overrides if self.context.job_id is None else None
else:
    input = inputs_with_overrides

# Always pass UiPathExecuteOptions explicitly for consistency with debug flow
options = UiPathExecuteOptions(resume=self.context.resume)
result = await execution_runtime.execute(input=input, options=options)

2. Resume Mode Validation

Added early validation in initiate_evaluation() to catch the unsupported scenario:

# Validate that resume mode is not used with multiple evaluations
if self.context.resume and len(evaluation_set.evaluations) > 1:
    raise ValueError(
        f"Resume mode is not supported with multiple evaluations. "
        f"Found {len(evaluation_set.evaluations)} evaluations in the set. "
        f"Please run with a single evaluation using --eval-ids to specify one evaluation."
    )

Benefits:

• ✅ Fails fast before any expensive operations (creating spans, loading evaluators, publishing events)
• ✅ Provides clear, actionable error message with guidance on how to fix
• ✅ Prevents confusing runtime behavior from the architectural limitation

Testing

• ✅ All existing tests pass (1807 tests)
• ✅ Added test_resume_with_multiple_evaluations_raises_error() to verify validation
• ✅ Created multiple-evals.json test fixture with 2 evaluations
• ✅ Functionally equivalent to previous behavior for single evaluations
• ✅ Makes explicit what was previously implicit
• ✅ Matches debug flow pattern

Impact

• Low risk: No functional behavior change for valid usage patterns
• High value:
  • Improves code consistency, maintainability, and explicitness
  • Prevents users from encountering confusing errors at runtime
  • Provides clear guidance when an unsupported scenario is attempted

[akshaylive]

How would this work when num_workers > 0? If it isn't supposed to work, we need to alert it somehow and halt execution. In other words, we may want to pass the resume to only the runtime instance that had previously suspended, and not to all runtime instances.

[Chibionos]

How would this work when num_workers > 0? If it isn't supposed to work, we need to alert it somehow and halt execution. In other words, we may want to pass the resume to only the runtime instance that had previously suspended, and not to all runtime instances.

It won't work for more than 1, its a limitation of suspend and resume. We can raise an error.

[akshaylive]

🚢