Scheduler Phase 2: VRAM-accounted admission + queue + eviction (the GPU arbiter both agents use)

[jaylfc]

Why now (Jay 2026-06-14)

@taos FLUX image-gen OOMd the shared 3060 because @taOSmd had ~9.4GB of Ollama models loaded. The A2A GPU-lease protocol (#893) is the stop-gap; the proper fix is the taOS resource queue both consumers submit to instead of negotiating by hand.

Current state (scaffolding exists)

• scheduling/resource_manager.py: discovers hardware, queries Ollama for loaded models, tracks VRAM/RAM pressure, adjusts job-queue concurrency. Awareness only, no enforcement.
• scheduler/scheduler.py + scheduler/resource.py: Phase 1, deliberately minimal -- synchronous routing, per-resource concurrency semaphore, first-resource-that-passes-admission runs. Priority queues, aging, preemption, max_wait_ms are Phase 2 and intentionally absent (per the module docstrings).
• scheduler/core_aware_scheduler.py (Reddit Client app (Knowledge Pipeline Step 3) #172): load_with_core_awareness(), not yet wired into the dispatch path.

The gap to build (Phase 2)

1. Per-device VRAM budget in resource.py: track total/used/free per accelerator (free = real-time, already accounts for external Ollama via the resource_manager read, so @taOSmd co-tenancy is visible).
2. Admission control: a GPU job (load model / generate / inference) is admitted only if est_vram <= free_vram; otherwise it queues.
3. Wait-queue with aging: queue blocked jobs, FIFO + aging to avoid starvation; max_wait_ms enforcement.
4. Eviction: before rejecting, evict idle keep-alive models (LifecycleManager) to make room.
5. Single arbiter for co-tenants: route both taOS backends AND (ideally) taOSmd's Ollama loads through admission. First step: taOS-side admission respects real-time free VRAM and queues/evicts; full mutual arbitration has taOSmd defer GPU loads to the same manager.
6. Wire TaskRouter.generate_image + core-aware loading (Reddit Client app (Knowledge Pipeline Step 3) #172) through the admission path.

Acceptance

Two consumers (image-gen + chat models) never co-load past a GPU's VRAM; over-budget work queues and runs when it fits (after eviction), with no OOM. The describe_image_capabilities tool reports live VRAM + queue depth. Subsumes the #893 A2A stop-gap. Brainstorm -> spec -> Phase 2 impl; do not block the current storybook demo on it.

[kilo-code-bot]

This issue appears to be a duplicate of #893.

Shared-hardware (GPU) coordination protocol between agents over A2A (#893)

Similarity score: 90%

This comment was generated by Kilo Auto-Triage.

[jaylfc]

Cluster lifecycle gap found wiring the 3060 (2026-06-14)

The decay/lifecycle engine already exists and works locally: LifecycleManager runs a local start_cmd + health-poll on demand and stop_cmd with graceful drain after keep_alive_minutes idle (0 = always-on, >0 = unload after N idle min, default 10); the image route starts a stopped auto_manage backend for the capability. Per-service decay = each backend/manifest's keep_alive_minutes.

The gap: it only manages backends the controller can start/stop directly (local start_cmd/stop_cmd). The 3060 sd-server is a remote container on linstation, so it's registered auto_manage: false and holds ~5GB VRAM resident forever, never auto-unloading — the GPU-sharing problem.

For cluster-wide auto load/unload, the lifecycle must be driven over the cluster via a worker agent on each box (the controller signals load/unload; the worker starts the container / loads the model, then unloads after the keep-alive decay to free VRAM). The timers/decay are done; what's missing is the remote drive. This sits at the intersection of:

• Scheduler Phase 2: VRAM-accounted admission + queue + eviction (the GPU arbiter both agents use) #894 (this) — admission decides WHEN eviction is needed,
• the LifecycleManager decay — decides the idle timeout,
• Worker/cluster-node service: auto-update with graceful pause, install, restart #890/Cluster worker: hybrid deployment (containerised default + native/bare-metal option) under one transport-agnostic contract #892 (worker contract) — carries the remote start/stop/drain.

Interim for the 3060: the #893 A2A lease protocol (manual claim/release). Acceptance should include: a remote GPU backend auto-unloads after its keep_alive so a co-tenant (e.g. @taOSmd Ollama) can reclaim the VRAM, with no manual coordination.