agents: rolling restart + session auto-resume across Pod restarts#527
Merged
jorisjonkers-dev-agents[bot] merged 4 commits intoJun 4, 2026
Merged
Conversation
Three gaps closed: 1. Rolling restart endpoint — `POST /api/v1/admin/runners/rolling-restart` gracefully scales down every active runner Pod (READY/STARTING/FAILED → IDLE, PVC preserved). The next session start on any workspace re-provisions a fresh Pod pulling `:latest`, picking up image or config changes without manual per-workspace intervention. 2. Claude session auto-resume — `AgentSessionManager.spawn()` now accepts `resumeCliSessionId`; when provided, Claude is launched with `--resume <id>` instead of `--session-id <newUUID>`. The conversation history lives on the shared credentials PVC and survives Pod restarts. `StartAgentSessionCommandHandler` looks up the latest non-failed Claude session for the workspace and forwards its `cliSessionId` so re-provisioned workspaces pick up where they left off automatically. 3. `RunnerMaintenanceService` owns the bulk scale-down logic; the controller is a thin wrapper. Scale-down failures are logged and skipped — the result counts only successful transitions. Node drain workflow: POST /api/v1/admin/runners/rolling-restart (pre-drain checkpoint) kubectl drain enschede-gtx-960m-1 --ignore-daemonsets <maintenance> kubectl uncordon enschede-gtx-960m-1 <workspaces auto-reprovision on next access, sessions resume>
Codex session files live in `~/.codex/sessions/` on the codex-credentials PVC and survive Pod restarts, but the gateway was not wiring up resume on re-provision (all previous session IDs for Codex are null in the DB). `AgentSessionManager.commandAndSessionIdFor()` now handles two Codex resume paths: - Sentinel `"LATEST"` → `codex resume --last`, which CWD-filters to the workspace's most recent session without requiring a stored session ID. - Specific UUID → `codex resume <id>` for direct targeting. `StartAgentSessionCommandHandler.previousSessionResumeId()` (renamed from `previousClaudeSessionId`) returns `"LATEST"` for CODEX when any prior non-failed session exists, and a specific UUID for CLAUDE. Two new tests cover both Codex resume paths.
Three gaps in the scale-down policy closed: 1. Connected-client lock — `ConnectedClientTracker` counts open browser WebSocket bridges per workspace. `IdleScaleDownScheduler` skips any workspace with at least one attached client, so closing a browser tab is the only way to disconnect, not a 30-min idle timer firing while you are watching. 2. AI-output activity — `SessionAttachHandler.UpstreamHandler` now calls `activity.touch(workspaceId)` on every frame received from the gateway (i.e., every byte the AI writes to the terminal). The idle timer resets as long as Claude/Codex is producing output, keeping the workspace alive throughout a long autonomous run even after the user disconnects. 3. Extended grace period for RUNNING sessions — when any agent session is in the RUNNING state the idle threshold switches from the normal 30 min to `agent-runtime.agent-idle-after-seconds` (default 4 h). Once the AI stops outputting and the 4-h window expires with no connected client, the workspace scales to zero. Summary of the combined policy: - Client connected → never scale down - AI outputting (client watching) → never scale down (touch resets timer) - AI outputting (client gone) → 4 h grace from last output - No AI, no client, 30 min quiet → scale to zero
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Five improvements to runner lifecycle, idle policy, and session continuity:
1. Rolling restart endpoint
POST /api/v1/admin/runners/rolling-restartgracefully scales down everyactive runner Pod (READY/STARTING/FAILED → IDLE, PVC preserved). On next
session start each workspace re-provisions, pulling
:latestand pickingup any image or config changes without manual per-workspace intervention.
Response:
{ "cycled": N, "workspaceIds": [...] }Node drain workflow:
2. Claude session auto-resume
Previously every
spawn()generated a fresh--session-id <uuid>, so are-provisioned Pod started a blank terminal even though conversation history
sat on the shared credentials PVC. The gateway now accepts
resumeCliSessionIdin the spawn request and launches Claude with
--resume <id>when provided.StartAgentSessionCommandHandlerlooks up the latest non-failed Claudesession's
cliSessionIdfor the workspace and forwards it automatically.3. Codex session auto-resume
Codex session files live in
~/.codex/sessions/(codex-credentials PVC)and survive Pod restarts. On re-provision
codex resume --lastis invokedwhich CWD-filters to the most recent session for that workspace — no stored
session ID required.
4. Idle policy: never evict connected users or active AI sessions
IdleScaleDownSchedulernow enforces three rules before scaling down:ConnectedClientTrackercounts open WebSocket bridgesper workspace. A workspace with any attached browser client is never scaled.
UpstreamHandler(gateway frames → browser)now calls
activity.touch()on every terminal frame. The 30-min idle timerresets as long as Claude/Codex is producing output, even after the user
disconnects.
RUNNINGagent sessionexists the idle threshold switches from 30 min to
agent-runtime.agent-idle-after-seconds(default 4 h). This protects longautonomous tasks that outlast the user's browser session.
Combined policy:
5. CI fixes
Instantimport (lint)AdminRunnerControllerannotated@Hidden(excluded from springdoc spec — not part of the frontend contract)StartAgentSessionCommandHandlerTestsessions mock changed torelaxed = trueSessionAttachHandlerTestupdated to pass newConnectedClientTrackerparamFiles changed
Dtos.kt,AgentSessionManager.kt,AgentController.ktAgentGatewayClient.ktStartAgentSessionCommandHandler.kt,RunnerMaintenanceService.kt(new),IdleScaleDownScheduler.kt,ConnectedClientTracker.kt(new)AdminRunnerController.kt(new,@Hidden),SessionAttachHandler.ktAgentSessionManagerTest.kt(+2),IdleScaleDownSchedulerTest.kt(+3),RunnerMaintenanceServiceTest.kt(new, 5 tests),StartAgentSessionCommandHandlerTest.kt,SessionAttachHandlerTest.kt