Heap-exhaustion diagnostics: /api/status instruments and resilient abort hook - #6
Merged
Conversation
The 2026-07-15 reboot recurrence was XS chunk-heap exhaustion after ~3.5 hours of match-day load (serial: "Chunk allocation ... failed" storm, then "XS abort: native stack overflow"). The abort hook could not persist the reason because building the string itself needs heap — lastAbort stayed empty. Now: - /api/status reports all instrumentation metrics (XS chunk/slot heap used, sockets, timers, promises) when the host bundles the instrumentation module; degrades to omitting the field elsewhere - the abort hook falls back to persisting the bare status string when the full reason cannot be allocated - companion host change (xymeow/stack-chan matchday-host 812d093) grows the XS arena from 432KB (57% full at boot) to 1.25MB in PSRAM Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under 1.6.0's concurrent HTTP handling, connection teardowns overlap inbound data far more than the old serialized handler did, reopening a window the arg-first NULL guard cannot cover: a tcpReceive already past the guard and mid-tail-walk when xs_tcp_destructor frees tcp->buffers/tcp. Observed as a LoadProhibited in tcpReceive (tcp.c:600) at ~25 min under light polling. The destructor now clears callbacks via the marshaled tcp_clear_callbacks_safe, which cannot run until an in-flight tcpReceive on the tcpip thread has returned. This revises the upstream note that the simpler arg-first variant was sufficient (it was, pre-concurrency). Also documents a separate, unfixed listener-accept race (pending socket RST before consumption) surfaced only by pathological connection bursts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ve race Deploying the marshaled destructor clear did not stop the crash: the device crashed again at 27 min under real load with a healthy heap, same tcpReceive backtrace. The fault address is a poison value (EXCVADDR=0xe2f61b44) with tcp itself valid, so the UAF is a receive-buffer node freed while still linked, not the tcp record my v5 change targeted. This is an unresolved SDK race across tcpReceive / xs_tcp_read / destructor on two cores; escalating to the maintainer with the poison-UAF evidence rather than guessing further. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observed live 2026-07-15: a remote-TTS stream settled neither onDone nor onError, leaving speakRemote's state.tts.busy stuck true, which made every later utterance return "tts busy" — commentary went silent until a reboot, with no crash. RemoteTTS.stream now arms a progress-reset watchdog (15s): armed before the first byte (so a dead TTS host is caught), reset on every onReady/onPlayed, and it tears the stream down and rejects if progress stops. All completion paths route through a single settle() that clears streaming and closes the streamer/audio first, so busy is released even if close throws. A healthy long utterance keeps resetting the timer and is never cut off. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tatic arena
The earlier note claimed a 1.25MB arena fixed the reboots; that edit was on
the wrong platform block (m5stack_core2) and did nothing. The real cause was
the CoreS3 chunk heap's incremental:0 hard-capping it at 256KB; the fix is
chunk { initial: 393216, incremental: 32768 } (stack-chan matchday-host
14d4d15), verified to grow past the old cap under load with zero aborts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #5, diagnosing last night's reboot recurrence.
Root cause (captured on serial): XS chunk-heap exhaustion after ~3.5 hours of match-day load —
lastAbortstayed empty because persisting the reason string itself needs heap. The 432KB XS arena was already 57% occupied at boot; the leak-or-watermark curve is being logged overnight to distinguish which.This PR (mod side, degrades gracefully on stock hosts)
/api/statusreports all instrumentation metrics (XS Chunk Heap Used,XS Slot Heap Used, sockets, timers, promises) underinstrumentswhen the host bundles the instrumentation module.docs/releases/1.6.0.md.Companion host change: xymeow/stack-chan
matchday-host812d093 grows the XS arena to 1.25MB (PSRAM-backed viaCONFIG_SPIRAM_USE_MALLOC) and bundles the instrumentation module. The instrumented build is already flashed and under observation.🤖 Generated with Claude Code