Skip to content

agents: rolling restart + session auto-resume across Pod restarts#527

Merged
jorisjonkers-dev-agents[bot] merged 4 commits into
mainfrom
agents/rolling-restart-and-session-resume
Jun 4, 2026
Merged

agents: rolling restart + session auto-resume across Pod restarts#527
jorisjonkers-dev-agents[bot] merged 4 commits into
mainfrom
agents/rolling-restart-and-session-resume

Conversation

@jorisjonkers-dev-agents

@jorisjonkers-dev-agents jorisjonkers-dev-agents Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

What changed

Five improvements to runner lifecycle, idle policy, and session continuity:

1. Rolling restart endpoint

POST /api/v1/admin/runners/rolling-restart gracefully scales down every
active runner Pod (READY/STARTING/FAILED → IDLE, PVC preserved). On next
session start each workspace re-provisions, pulling :latest and picking
up any image or config changes without manual per-workspace intervention.

Response: { "cycled": N, "workspaceIds": [...] }

Node drain workflow:

# Checkpoint all runners before the drain
curl -X POST -H "X-User-Id: $UID" https://assistant.<domain>/api/v1/admin/runners/rolling-restart
kubectl drain enschede-gtx-960m-1 --ignore-daemonsets --delete-emptydir-data
# <maintenance>
kubectl uncordon enschede-gtx-960m-1
# Workspaces auto-reprovision on next access, sessions resume automatically

2. Claude session auto-resume

Previously every spawn() generated a fresh --session-id <uuid>, so a
re-provisioned Pod started a blank terminal even though conversation history
sat on the shared credentials PVC. The gateway now accepts resumeCliSessionId
in the spawn request and launches Claude with --resume <id> when provided.
StartAgentSessionCommandHandler looks up the latest non-failed Claude
session's cliSessionId for the workspace and forwards it automatically.

3. Codex session auto-resume

Codex session files live in ~/.codex/sessions/ (codex-credentials PVC)
and survive Pod restarts. On re-provision codex resume --last is invoked
which CWD-filters to the most recent session for that workspace — no stored
session ID required.

4. Idle policy: never evict connected users or active AI sessions

IdleScaleDownScheduler now enforces three rules before scaling down:

  • Connected client: ConnectedClientTracker counts open WebSocket bridges
    per workspace. A workspace with any attached browser client is never scaled.
  • AI output → activity touch: UpstreamHandler (gateway frames → browser)
    now calls activity.touch() on every terminal frame. The 30-min idle timer
    resets as long as Claude/Codex is producing output, even after the user
    disconnects.
  • Running sessions use 4-h grace period: when any RUNNING agent session
    exists the idle threshold switches from 30 min to
    agent-runtime.agent-idle-after-seconds (default 4 h). This protects long
    autonomous tasks that outlast the user's browser session.

Combined policy:

client connected                → never scale down
AI outputting (client watching) → never scale down
AI outputting (client gone)     → 4 h grace from last output
no AI, no client, 30 min quiet  → scale to zero

5. CI fixes

  • Removed unused Instant import (lint)
  • AdminRunnerController annotated @Hidden (excluded from springdoc spec — not part of the frontend contract)
  • StartAgentSessionCommandHandlerTest sessions mock changed to relaxed = true
  • SessionAttachHandlerTest updated to pass new ConnectedClientTracker param

Files changed

Area Files
agent-gateway Dtos.kt, AgentSessionManager.kt, AgentController.kt
assistant-api domain AgentGatewayClient.kt
assistant-api application StartAgentSessionCommandHandler.kt, RunnerMaintenanceService.kt (new), IdleScaleDownScheduler.kt, ConnectedClientTracker.kt (new)
assistant-api web AdminRunnerController.kt (new, @Hidden), SessionAttachHandler.kt
Tests AgentSessionManagerTest.kt (+2), IdleScaleDownSchedulerTest.kt (+3), RunnerMaintenanceServiceTest.kt (new, 5 tests), StartAgentSessionCommandHandlerTest.kt, SessionAttachHandlerTest.kt

Three gaps closed:

1. Rolling restart endpoint — `POST /api/v1/admin/runners/rolling-restart`
   gracefully scales down every active runner Pod (READY/STARTING/FAILED →
   IDLE, PVC preserved). The next session start on any workspace
   re-provisions a fresh Pod pulling `:latest`, picking up image or
   config changes without manual per-workspace intervention.

2. Claude session auto-resume — `AgentSessionManager.spawn()` now accepts
   `resumeCliSessionId`; when provided, Claude is launched with
   `--resume <id>` instead of `--session-id <newUUID>`. The conversation
   history lives on the shared credentials PVC and survives Pod restarts.
   `StartAgentSessionCommandHandler` looks up the latest non-failed
   Claude session for the workspace and forwards its `cliSessionId` so
   re-provisioned workspaces pick up where they left off automatically.

3. `RunnerMaintenanceService` owns the bulk scale-down logic; the
   controller is a thin wrapper. Scale-down failures are logged and
   skipped — the result counts only successful transitions.

Node drain workflow:
  POST /api/v1/admin/runners/rolling-restart   (pre-drain checkpoint)
  kubectl drain enschede-gtx-960m-1 --ignore-daemonsets
  <maintenance>
  kubectl uncordon enschede-gtx-960m-1
  <workspaces auto-reprovision on next access, sessions resume>
@jorisjonkers-dev-agents jorisjonkers-dev-agents Bot added the enhancement New feature or request label Jun 4, 2026
Codex session files live in `~/.codex/sessions/` on the codex-credentials
PVC and survive Pod restarts, but the gateway was not wiring up resume on
re-provision (all previous session IDs for Codex are null in the DB).

`AgentSessionManager.commandAndSessionIdFor()` now handles two Codex resume
paths:
- Sentinel `"LATEST"` → `codex resume --last`, which CWD-filters to the
  workspace's most recent session without requiring a stored session ID.
- Specific UUID → `codex resume <id>` for direct targeting.

`StartAgentSessionCommandHandler.previousSessionResumeId()` (renamed from
`previousClaudeSessionId`) returns `"LATEST"` for CODEX when any prior
non-failed session exists, and a specific UUID for CLAUDE.

Two new tests cover both Codex resume paths.
Three gaps in the scale-down policy closed:

1. Connected-client lock — `ConnectedClientTracker` counts open browser
   WebSocket bridges per workspace. `IdleScaleDownScheduler` skips any
   workspace with at least one attached client, so closing a browser tab
   is the only way to disconnect, not a 30-min idle timer firing while
   you are watching.

2. AI-output activity — `SessionAttachHandler.UpstreamHandler` now calls
   `activity.touch(workspaceId)` on every frame received from the gateway
   (i.e., every byte the AI writes to the terminal). The idle timer resets
   as long as Claude/Codex is producing output, keeping the workspace alive
   throughout a long autonomous run even after the user disconnects.

3. Extended grace period for RUNNING sessions — when any agent session is
   in the RUNNING state the idle threshold switches from the normal 30 min
   to `agent-runtime.agent-idle-after-seconds` (default 4 h). Once the AI
   stops outputting and the 4-h window expires with no connected client,
   the workspace scales to zero.

Summary of the combined policy:
  - Client connected                → never scale down
  - AI outputting (client watching) → never scale down (touch resets timer)
  - AI outputting (client gone)     → 4 h grace from last output
  - No AI, no client, 30 min quiet  → scale to zero
@jorisjonkers-dev-agents jorisjonkers-dev-agents Bot merged commit 089eb46 into main Jun 4, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant