[Bounty #3926] fix(runtime): avoid race when resuming paused agents — resume runtime

[neuralmint]

Summary

Adds a durable state machine guard to prevent race conditions when resuming paused agents. This ensures that concurrent resume operations are serialised and that invalid state transitions are rejected before they can cause duplicate work or inconsistent run state.

Changes

AgentRuntime (src/agent/runtime.py)

• Added RuntimeState.PAUSED and a _VALID_RUNTIME_TRANSITIONS matrix enumerating all legal state transitions
• Added threading.Lock around all state mutations (start, stop, resume, pause, get_state)
• Added resume() method — only valid when current state is PAUSED; raises RuntimeStateError otherwise
• Added pause() method — only valid when current state is RUNNING
• Both _transition() and resume() validate the transition before committing

AgentRegistry (src/agent/registry.py)

• Added AgentStatusError exception for invalid lifecycle transitions
• Added _VALID_STATUS_TRANSITIONS matrix for the six agent statuses (PENDING, RUNNING, PAUSED, STOPPED, FAILED, TERMINATED)
• update_status() now raises AgentStatusError on invalid transitions (e.g. PAUSED → PENDING is rejected)
• Same-status transitions are allowed (idempotent no-op)

OrchestrationEngine (src/orchestrator/engine.py)

• Added resume_agent() method that validates both the registry status (must be PAUSED) and runtime state (must be PAUSED) before committing the transition to RUNNING
• Raises ResumeStateError when guard conditions are not met

Tests

• test_agent_runtime.py: 22 new tests covering:
  • Basic state machine transitions (start, stop, pause, resume)
  • Invalid transitions rejected (resume from RUNNING/STOPPED, pause from PAUSED)
  • Concurrent resume: 5 threads racing — exactly one succeeds, the rest are rejected
  • Concurrent resume + pause on different agents: no deadlock
  • State consistency under concurrent reads/writes
  • Parametrized transition matrix covering 12 valid/invalid paths
  • Engine-level resume_agent guard tests

Test Output

tests/test_agent_runtime.py     ✓ 22 passed
tests/test_agent_registry.py    ✓ 13 passed (3 new guard tests)
tests/test_compensation.py      ✓ 27 passed (unchanged)
tests/test_scheduler.py         ✓ 5 passed (unchanged)
tests/test_config.py            ✓ 5 passed (unchanged)

Closes #3926