spec+plan: STT readiness tells the truth about CPU fallback (#155)

[OriNachum]

Spec + plan for issue #155. Docs only — no code changes. The implementation is the plan's ten tasks, not this PR.

What #155 reported

On spark, the Parakeet stt gear served POST /v1/audio/transcriptions with 200 while its own /v1/health/ready returned 503 on a failing CUDA probe. Docker marked it unhealthy, the gateway refused to route, and lobes capabilities said loaded: yes while lobes fleet status said unhealthy. The reporter could not tell whether the model was serving on CPU or whether the 200s were hollow, and had no speech sample to find out.

The unknown was settled empirically, not inferred

The reporting host was reachable, so this was resolved by measurement rather than reasoning. The STT container's own base image ships known-content speech — /usr/share/sounds/alsa/Front_Center.wav says "front center":

POST 127.0.0.1:9002/v1/audio/transcriptions  →  {"text":"Front, center."}   1.68s
nvidia-smi --query-compute-apps               →  no stt process on the GPU

Correct transcript, zero GPU memory. Reading 1: Parakeet was serving genuinely useful transcripts on CPU.

Root cause is outside lobes

docker exec model-gear-stt nvidia-smi -L     → Failed to initialize NVML: Unknown Error
docker inspect model-gear-stt                → valid nvidia DeviceRequest, NVIDIA_VISIBLE_DEVICES=all
docker run --rm --gpus all nvidia/cuda ...   → GPU 0: NVIDIA GB10        ← fresh container is fine

Both long-running audio containers had lost their device cgroup mid-life (systemd reloads at 16:52/20:32). vLLM containers survived because they already held a CUDA context. A restart restored the lane in ~20s. The fresh-container check is the generalizable diagnostic: host nvidia-smi alone cannot distinguish "this container lost the GPU" from "the host GPU is broken."

lobes does not own repairing the container runtime — but it does own reporting it honestly, which is what the spec covers.

Four lobes-side defects (none fixed by the restart)

Surface	Defect
_readiness.py (×2 copies)	two-state by construction — CPU-serving is indistinguishable from dead
roles.py	CLI passes audio_ready=None → echoes config, so capabilities contradicts fleet status
gateway/server.py	503 says "warming up — retry shortly" for a five-hour state → unbounded caller retry
audio_facade.py	aggregate_audio_ready ANDs tts+stt, so either lane takes both down

Decisions recorded in the frame

• c17 — readiness is device-agnostic: a loaded model is ready on any device, and every audio lane routes. Residual risk accepted and recorded (r2): a CPU-serving stt makes the realtime session ~1.1× realtime.
• c18 — the batch facade splits per-lane, so neither lane is hostage to the other.
• c21 — /v1/realtime keeps a composite verdict, because the bridge wraps stt + tts + VAD + generate.
• c22 — that composite is itself incomplete today: app.py's ready() probes only tts_url and stt_url, never VAD and never the generate lane — which on a mesh box may be a proxied peer. Surfaced while acting on c21.

Two claims I proposed early (a "degraded rung" design and a matching boundary) were rejected once they contradicted c17, and replaced by c19/c20 rather than quietly reworded.

Both secondary observations in the issue are false alarms

• The bare python3 healthcheck is correct — Dockerfile.parakeet is FROM scitrera/dgx-spark-vllm, which ships /usr/bin/python3. The adjacent warning comment is about Chatterbox's different nvidia/cuda base.
• GET /v1/realtime 404 is version skew — the deployed gateway is lobes 0.52.3; the route shipped in 0.53.0. Current code refuses a plain GET with 426 not_an_upgrade.

The plan

10 tasks, 5 waves, all 28 coverage targets claimed. Every wave is file-disjoint and each task carries an explicit FILES: instruction, so parallel execution yields the same result as serial:

w0  t1 _readiness.py ×2            t3 audio_facade.py
w1  t2 listen_server.py            t4 app.py            t9 docs
w2  t5 gateway/server.py           t6 roles.py + capabilities.py
w3  t7 probe script                t8 new test module
w4  t10 live run + evidence        [operator-only]

t10 is deliberately split from t7: a subagent in a throwaway worktree cannot force a container to CPU or capture a real transcript, and per #108 no doc may claim validation until that transcript lands under docs/evidence/.

Three risks recorded rather than smoothed over:

• r1 — t4 puts a cross-box peer call on a health path; needs a bounded/cached probe, not a synchronous dial.
• r2 — device-agnostic readiness makes a real GPU regression invisible to docker health; the reported device becomes the only signal, so it must surface where an operator looks.
• r3 — the readiness code ships inside the stt image, so nothing reaches a live box without a rebuild + MODEL_GEAR_VERSION bump.

Files

• docs/specs/2026-07-24-stt-readiness-truth-155.md
• docs/plans/2026-07-24-stt-readiness-truth-155.md
• .devague/frames/ + .devague/plans/ state (the evidence trail for every confirmed claim)

Note for review

agex pr lint reports one user-dotfile-reference violation at CHANGELOG.md:978. It is pre-existing (introduced in 3dba470, #61), not part of this diff, and is a historical changelog entry describing a real path — I did not rewrite it.

Closes #155 only in the sense of diagnosing it; the defects remain open until the plan is built.

• lobes (Claude)

[OriNachum]

/agentic_review

[qodo-code-review]

PR Summary by Qodo

Add spec+plan for STT readiness truth about CPU fallback (#155)

📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add spec and build plan for issue #155 STT readiness contradictions under CUDA loss.
• Capture devague frame/plan exports documenting decisions, risks, and acceptance criteria.
• Bump project version to 0.54.2 and record the change in CHANGELOG.

Diagram

graph TD
  client{{"Client"}} --> gw["Gateway"] --> route{"Audio lane?"}
  route -->|"/v1/audio/transcriptions"| stt["STT gear"]
  route -->|"/v1/audio/speech"| tts["TTS gear"]
  gw --> rt["Realtime bridge"]
  dock{{"Docker health"}} --> stt --> ready["/v1/health/ready"]
  dock --> tts
  cli["lobes CLI"] --> gw
  subgraph Legend
    direction LR
    _ext{{"External"}} ~~~ _svc["Service"] ~~~ _dec{"Decision"}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Publish only rendered markdown (omit devague JSON exports)

• ➕ Much smaller diffs; easier human review
• ➕ Avoids committing tool-internal state that may churn
• ➖ Loses machine-readable traceability from frame→plan→tasks
• ➖ Harder to regenerate/validate plan coverage against claims later

2. Split into two PRs: spec first, plan second

• ➕ Reduces review scope per PR
• ➕ Lets stakeholders approve problem statement/decisions before task breakdown
• ➖ Creates more coordination overhead
• ➖ Plan review depends on spec details anyway; likely reviewed together

3. Include a minimal implementation patch for one defect (e.g., gateway 503 message)

• ➕ Immediately reduces user harm (misleading 'retry shortly')
• ➕ Validates the spec’s direction with a small, testable change
• ➖ Breaks the stated 'docs-only' intent for this PR
• ➖ Risks partial fixes without the full per-lane + readiness contract work

Recommendation: Given the explicit intent (“docs only — no code changes”), the current approach is reasonable: land the spec+plan as the shared contract, then implement as follow-on tasks. If reviewer bandwidth is limited, consider omitting devague exports in future docs-only PRs, but keeping them here is defensible because the plan’s acceptance criteria and coverage mapping are central to coordinating the multi-surface fix.

Files changed (9) +1107 / -13

Documentation (7) +1105 / -11

stt-readiness-truth-155.jsonAdd devague frame export for issue #155 readiness truth +477/-0

Add devague frame export for issue #155 readiness truth

• Introduces the exported frame capturing the problem statement, measured evidence, decisions, requirements, and honesty conditions for STT readiness under CPU fallback.

.devague/frames/stt-readiness-truth-155.json

stt-readiness-truth-155.jsonAdd devague plan export with 10 implementation tasks +389/-0

Add devague plan export with 10 implementation tasks

• Adds the exported build plan mapping tasks (t1–t10) to claims/honesty conditions, including per-file instructions and acceptance criteria.

.devague/plans/stt-readiness-truth-155.json

colleague__public.jsonlRefresh eidetic colleague public memory record metadata +1/-1

Refresh eidetic colleague public memory record metadata

• Updates a stored memory entry (e.g., last_recall/recall_count) without changing the underlying work-lesson content intent.

.eidetic/memory/colleague__public.jsonl

lobes__public.jsonlRefresh eidetic lobes public memory record metadata +13/-10

Refresh eidetic lobes public memory record metadata

• Updates several stored memory entries’ recall metadata (last_recall/recall_count) to reflect recent access; no functional code impact.

.eidetic/memory/lobes__public.jsonl

CHANGELOG.mdAdd 0.54.2 changelog entry referencing the #155 spec/plan +6/-0

Add 0.54.2 changelog entry referencing the #155 spec/plan

• Documents the new spec and plan deliverables for issue #155 under the 0.54.2 release header.

CHANGELOG.md

2026-07-24-stt-readiness-truth-155.mdPublish build plan markdown for STT readiness truth (#155) +120/-0

Publish build plan markdown for STT readiness truth (#155)

• Adds the human-readable plan with tasks, dependencies, acceptance criteria, and risks for implementing device-agnostic readiness and per-lane audio gating.

docs/plans/2026-07-24-stt-readiness-truth-155.md

2026-07-24-stt-readiness-truth-155.mdPublish spec markdown describing the readiness/routing truth contract +99/-0

Publish spec markdown describing the readiness/routing truth contract

• Adds the spec describing the observed contradiction, the desired after-state, explicit boundaries/non-goals, and testable honesty conditions and success signals.

docs/specs/2026-07-24-stt-readiness-truth-155.md

Other (2) +2 / -2

currentPoint devague current workspace at stt-readiness-truth-155 +1/-1

Point devague current workspace at stt-readiness-truth-155

• Updates the devague pointer to make stt-readiness-truth-155 the active spec/plan context.

.devague/current

pyproject.tomlBump lobes-cli version to 0.54.2 +1/-1

Bump lobes-cli version to 0.54.2

• Increments the project version to reflect the new spec/plan release entry.

pyproject.toml

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 80 rules

✅ Skills: doc-test-alignment, cicd

1. Stale current_plan pointer 🐞 Bug ⚙ Maintainability

Description

.devague/current was updated to the new spec frame slug, but .devague/current_plan still points
to the previous plan slug, leaving the repo’s devague “current” pointers internally inconsistent.

Code

.devague/current[1]

+stt-readiness-truth-155

Relevance

⭐⭐⭐ High

Simple deterministic consistency fix for devague pointers; low risk and aligns with docs-only
intent.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The branch sets the active frame to stt-readiness-truth-155 but the active plan pointer remains
the prior slug, despite the new plan file being present.

.devague/current[1-1]
.devague/current_plan[1-1]
.devague/plans/stt-readiness-truth-155.json[1-6]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`.devague/current` now points at `stt-readiness-truth-155`, but `.devague/current_plan` still points at `realtime-voice-to-voice-astro-test-site-151`. This makes the repo’s devague “current” frame/plan pointers inconsistent.

## Issue Context
A matching plan file for the new slug exists under `.devague/plans/`, so the stale pointer is likely accidental.

## Fix Focus Areas
- .devague/current[1-1]
- .devague/current_plan[1-1]
- .devague/plans/stt-readiness-truth-155.json[1-6]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. uv.lock version mismatch 🐞 Bug ☼ Reliability

Description

pyproject.toml bumps lobes-cli to 0.54.2, but uv.lock still records the editable package as
0.54.1, which can cause lockfile staleness/validation issues and surprise diffs during uv
operations.

Code

pyproject.toml[3]

+version = "0.54.2"

Relevance

⭐⭐⭐ High

Team has accepted version/source-of-truth consistency fixes; lockfile staleness is a common
reliability footgun.

PR-#45

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The project metadata reports version 0.54.2, but the lockfile’s editable lobes-cli package stanza
still pins 0.54.1.

pyproject.toml[1-4]
uv.lock[441-445]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The project version was bumped in `pyproject.toml`, but the editable package entry in `uv.lock` still contains the previous version. This makes the lockfile inconsistent with the project metadata.

## Issue Context
`uv.lock` contains a `[[package]]` stanza for the editable root project.

## Fix Focus Areas
- pyproject.toml[1-4]
- uv.lock[441-445]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

1. Stale current_plan pointer 🐞 Bug ⚙ Maintainability

.devague/current was updated to the new spec frame slug, but .devague/current_plan still points
to the previous plan slug, leaving the repo’s devague “current” pointers internally inconsistent.

Agent Prompt

## Issue description
`.devague/current` now points at `stt-readiness-truth-155`, but `.devague/current_plan` still points at `realtime-voice-to-voice-astro-test-site-151`. This makes the repo’s devague “current” frame/plan pointers inconsistent.

## Issue Context
A matching plan file for the new slug exists under `.devague/plans/`, so the stale pointer is likely accidental.

## Fix Focus Areas
- .devague/current[1-1]
- .devague/current_plan[1-1]
- .devague/plans/stt-readiness-truth-155.json[1-6]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

2. Uv.lock version mismatch 🐞 Bug ☼ Reliability

pyproject.toml bumps lobes-cli to 0.54.2, but uv.lock still records the editable package as
0.54.1, which can cause lockfile staleness/validation issues and surprise diffs during uv
operations.

Agent Prompt

## Issue description
The project version was bumped in `pyproject.toml`, but the editable package entry in `uv.lock` still contains the previous version. This makes the lockfile inconsistent with the project metadata.

## Issue Context
`uv.lock` contains a `[[package]]` stanza for the editable root project.

## Fix Focus Areas
- pyproject.toml[1-4]
- uv.lock[441-445]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools