fix(telegram): reap abandoned publication staging temps - #3614
Conversation
31f78f1 to
4207eb4
Compare
The notification self-heal reaper never claimed `.tmp` staging files, so every failed publication left one unreachable file in the agent notifications directory permanently. `writeJsonAtomic` stages a sibling `<name>.<pid>.<epoch-ms>.<suffix>.tmp` and renames it over the destination. If the staging write or the rename fails, or the process dies between the two, that temp is never published and never read again. No prefix in NOTIFICATION_LEAK_ARTIFACT_PREFIXES matched it, so the reaper walked past it. This accumulates across the roots registry, daemon state, callback aliases, seen-update ids, and the topic registry snapshot, and is most visible where a rename-blocking condition persists (Windows EPERM from an antivirus or indexer handle, EACCES, EIO, ENOSPC). Reaping is shape-matched and still bounded by the existing five-minute mtime grace window, so a temp an in-flight publication is still staging is never removed. Fixing it in the reaper rather than the writer also reclaims temps orphaned by a crash, which no writer-side unwind can reach, and leaves the protected `writeJsonAtomic` lifecycle declaration byte-identical so no DAEMON_GENERATION bump is required.
4207eb4 to
03b5768
Compare
|
Rebased onto current dev A note on scope, since it changed after the first push. My initial version unwound the staged temp inside Reaping is shape-matched and still bounded by the existing five-minute mtime grace window, so a temp an in-flight publication is still staging is never removed; one of the three tests pins exactly that. Ready whenever you have time to look. |
Yeachan-Heo
left a comment
There was a problem hiding this comment.
Exact-head hostile review: REQUEST_CHANGES
Reviewed head: 03b57684b9a94efd196166b610ee0f219fb01800
Exact/current base: 7f5167c864f938558840084ad218d48132f0a67a
P0: 0. P1: 2.
-
The shape+mtime reaper is not publisher-generation/liveness bound. A live or blocked
writeJsonAtomicpublisher can remain staged past five minutes; startup/periodic reaping cannot distinguish it from crash residue and can delete its in-flight file. Encode a finite destination plus publisher generation/acquisition identity and reap only proven-dead claims; this protected writer change requires a daemon generation bump and guard regeneration. -
Cleanup is pathname-only
statthen awaitedunlink.statfollows symlinks and no regular-file/single-link identity is bound through deletion, so a readdir/stat→replacement ABA can delete a fresh generation's staged file; hardlinks, reparse points, and Windows busy-file semantics are unfenced. Use no-follow identity capture plus an identity-bound removal primitive and cover replacement/symlink/hardlink/Windows sharing races.
Exact Actions evidence is terminal green (17 success / 5 skipped / 0 bad), including the targeted test and Windows Telegram safety, but does not cover these authority races. The required repair overlaps #3596's exact native/generation authority and must be coordinated rather than merged as a shape-only cleanup.
—
[repo owner's gaebal-gajae (clawdbot) 🦞]
Hello, and thank you for the review time on this one.
What this fixes
The Telegram notification self-heal reaper never claimed
.tmpstaging files, so every failed publication left one unreachable file in the agentnotifications/directory permanently.writeJsonAtomicstages a sibling<name>.<pid>.<epoch-ms>.<suffix>.tmpand renames it over the destination. If the staging write or the rename fails, or the process dies between the two, that temp is never published and never read again. No prefix inNOTIFICATION_LEAK_ARTIFACT_PREFIXESmatched it, soreapStaleNotificationArtifacts— the one thing that exists to clean this directory — walked straight past it.Reachable today on
devthrough the publicregisterNotificationRootentry point, and equally through daemon ownership state, callback aliases, seen-update ids, and the topic registry snapshot. Most visible where a rename-blocking condition persists: a WindowsEPERMfrom an antivirus or indexer holding a handle,EACCES,EIO,ENOSPC.Why the reaper, not the writer
I first wrote this as a
try/catchunwind insidewriteJsonAtomic. Your generation guard correctly rejected that:writeJsonAtomicis a protected lifecycle declaration and the change demanded a strictly higherDAEMON_GENERATION, which lives intelegram-daemon-contract.ts— owned by the active #3596. I reverted it rather than contend for that file.Fixing it in the reaper turned out to be strictly better anyway:
writeJsonAtomicstays byte-identical todev, so noDAEMON_GENERATIONbump and no manifest edit. The guard passes unchanged, andscripts/telegram-daemon-generation-manifest.jsonis untouched.Safety
Reaping is shape-matched (
/^.+\.\d+\.\d+\.[0-9a-z]+\.tmp$/) and still bounded by the existing five-minute mtime grace window, so a temp that an in-flight publication is still staging is never removed. A published notification file is never matched.Verification
Reproduced the defect before fixing it, on this host (darwin arm64):
On unmodified
devsource — 5/5 runs, deterministic:With the fix — 8/8 runs, deterministic:
I ran it eight times each way on purpose: my first draft of the staleness assertion was itself flaky (with
graceMs: 0, a temp written in the same millisecond can carry a fractionalmtimeMsslightly ahead of an integerDate.now(), reading as negative age and being skipped as still-staging). The test now advances the reaper's clock past the grace window instead, which is deterministic.Also green:
notifications-telegram-daemon.test.ts+-2960+-2960-redteam+-self-heal+ this file — 575 pass / 0 failtelegram-daemon-generation-guard --validate-current-tree— exit 0telegram-daemon-generation-guard.test.ts— 41 pass / 0 failtsc -p tsconfig.json --noEmit— exit 0biome check— exit 0Rebased onto the current
devtip. Three files: the reaper rule, one regression test, one changelog bullet. No native bindings, and nothing in #3596's file set.Happy to adjust anything — please take whatever time you need.