Repair the trust store for real, and two diagnostics that answer today's open reports - #601
Merged
Merged
Conversation
… what it was doing A SoundTouch 10 owner reports that his speaker reboots itself every night and that the first hardware preset press next morning lights the orange LED and plays nothing (#600). His two bundles show the press reaching the speaker correctly: it selects slot 5, STR logs that the box activates the native radio preset itself, and 700 ms later the speaker reports 4502 BMX_JSON_PARSE_ERROR and drops to INVALID_SOURCE. Before the nightly reboot the same speaker resolved a station through STR normally. What the bundle cannot say is what the speaker fetched, or from where. A native radio preset carries a location RELATIVE to the baseUrl in the BMX service list, so a speaker that never fetches that list cannot resolve a press at all, and the failure then surfaces as a parse error that reads like a broken station. That the request trail was empty is suggestive, but it is inference, and inference is what this has been answered with twice. Two facts, both cheap, both absent until now: Every error the speaker reports is kept alongside the preset location it was acting on and how long after the selection it arrived. The log already carried the errors, but finding them by eye among thousands of lines and pairing them with a press by timestamp is exactly the work a bundle should have done already. And the BMX registry response is counted. Zero fetches since the agent started is a statement, not a hint: the speaker cannot have resolved anything relative to a baseUrl it never asked for. Neither changes behaviour. The next bundle from this reporter should say outright which of the two mechanisms is at work. Refs #600 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ft broken, and takes effect The repair shipped in v0.9.43 did not help the speaker it was written for. Its owner updated, pressed the key, and got the same silence, and his diagnostic says why in two lines: store /etc/pki/tls/certs/ca-bundle.crt roots=2 publicRootsMissing=false repair /etc/pki/tls/certs/ca-bundle.crt healthy 1 -> 1 Two mistakes, both here. The test was "more than zero public roots". His bundle held exactly two certificates, STR's root plus a single survivor, so one public root counted as healthy and the file was left alone. A working speaker of the same firmware carries 158 and 165. A store down to single digits is the same corruption caught one step later, and calling it healthy is how a broken speaker gets told it is fine. The threshold is now a believable NUMBER of roots. The second is worse, because it applied even where the repair did fire. Go reads the system trust store once per process and caches it. Both the agent and the Spotify engine are started by a boot script that has already mounted the broken overlay, so both cached the poisoned copy long before the repair touched the file, and nothing re-reads it. Repairing the file changed what was on disk and nothing about what the speaker could reach. So a repair that actually rebuilt a store now exits, and the boot script's watchdog brings the agent and the engine back on the fixed store. Guarded to once per boot: a restart that keeps repeating is worse than a speaker that trusts nothing, and this path only runs on a speaker that already trusts nothing, so there is no playback to interrupt. Reproduced on an ST10 in the reporter's exact shape, one surviving root beside ours: 13:05:16 trust store lost its public roots, repairing certificates=2 13:05:16 trust store repaired publicRoots=166 firmwareRoots=166 13:05:16 restarting so this process and the Spotify engine read the fixed store 13:05:31 streborn starting Afterwards the store holds 167 and a handshake through it succeeds, where before the repair called the same file healthy and left it at two. Release-Note: fix(speaker): a speaker that trusts no internet station is now really repaired, including the case the previous version reported as healthy Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… looked like first A live Portable lost its agent mid-Spotify on 2026-08-14: a healthy session (156 s attached, 3049 KB forwarded), then nothing for 76 seconds, then "streborn starting" with bootReason="agent-respawn (box already up 2h19m)". Spotify never recovered afterwards, and nothing in the log could say whether the process had panicked, been killed for memory, or been stopped on purpose. One thing it could already say, by omission: a deliberate stop logs "shutdown signal received" from the SIGTERM handler and there was no such line, so the process was killed rather than asked to stop. That is reasoning from an absence, which is what this replaces. Two measurements. A heartbeat every 30 s on tmpfs, so the next start knows what the previous run looked like just before it died: free memory, the agent's own footprint, thread count and whether Spotify was streaming. It costs no NAND, and it disappears on a box reboot while surviving an agent respawn, which is the same distinction bootReason already draws. And the kernel's own verdict: the OOM killer names its victim, and that line settles "killed for memory" outright instead of leaving it to be inferred from a memory graph. Both are reported as `last_exit` in /api/debug/state, and the start logs them only when there is something to say, so a normal boot stays silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JRpersonal
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes from today's reports. One is a genuine fix and corrects something
v0.9.43 shipped; the other two are instrumentation with no behaviour change.
The certificate repair in v0.9.43 does not work
It did not help the speaker it was written for. The owner updated, pressed the
key, got the same silence, and his diagnostic says why in two lines:
Two mistakes, both mine.
The threshold was "more than zero". His bundle held two certificates, STR's
root plus one survivor, so a single public root counted as healthy and the file
was skipped. A working speaker of the same firmware carries 158 and 165.
And a repaired file changes nothing by itself. Go reads the system trust
store once per process and caches it. The agent and the Spotify engine are both
started by a boot script that has already mounted the broken overlay, so both
cached the poisoned copy before the repair ran, and nothing re-reads it. A
repair that actually rebuilds a store now exits so the watchdog brings both
back on the fixed store, guarded to once per boot.
Reproduced on an ST10 in his exact shape:
Store afterwards: 167 roots, and a handshake through it succeeds.
What the speaker complained about (#600)
A hardware preset press dies on
4502 BMX_JSON_PARSE_ERRORand the bundlecannot say what the speaker fetched. Box errors are now kept with the preset
location the box was acting on, and the BMX registry response is counted, so
registryFetches: 0becomes a statement rather than an inference from an emptyrequest trail.
Why the agent was killed
A Portable lost its agent mid-Spotify with no trace beyond the gap in the log.
A 30 s heartbeat on tmpfs now records what the previous run looked like before
it died, and the start scans the kernel ring buffer for the OOM killer's own
verdict. Reported as
last_exit.Refs #600
🤖 Generated with Claude Code