fix(analytics): keep scrapeServerIfStale to its "never throws" contract - #598
Merged
Merged
Conversation
The outer `try` in `scrapeServerIfStale` had a `finally` but no `catch`, and the
two awaits that talk to the cache — `cacheStore()` and `store.get(key)` — sit
ABOVE the inner try that guards `scrapeServer`. A cache backend error (Redis
down, connection reset) therefore rejected the returned promise, which two
docstrings promise cannot happen: this function's own ("self-throttles and never
throws") and `runAnalyticsScrapeSweep`'s ("Never throws — scrapeServerIfStale
swallows per-server failures, so one unreachable box can't fail the job or stop
the remaining servers").
Two consequences followed from that:
- The three fire-and-forget read handlers — geo.service.ts:221 and
analytics.controller.ts:239/272 — call it as `void scrapeServerIfStale(id)`.
A rejected voided promise is an unhandled rejection, which is fatal on Node's
default `--unhandled-rejections=throw`: a Redis blip while someone opens the
analytics tab takes the API process down.
- The sweep awaits it in a loop with no try/catch, so the same error abandoned
every server after the failing one — exactly the "one box can't stop the
remaining servers" guarantee it documents.
Catches at the outer level and logs through the same `debug` channel the inner
handler uses. No behaviour change on the success or throttled paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #597.
scrapeServerIfStale's outertryhad afinallybut nocatch, and the two awaits that talk to the cache —cacheStore()andstore.get(key)— sit above the inner try that guardsscrapeServer. A cache backend error therefore rejected the returned promise, which two docstrings promise cannot happen (this function's own, andrunAnalyticsScrapeSweep's).Consequences, both in #597 in detail:
void scrapeServerIfStale(...)read handlers turn that rejection into an unhandled rejection — fatal under Node's default--unhandled-rejections=throw. A Redis blip while the analytics tab is open takes the API process down.awaitin aforloop abandons every server after the failing one, and skips the collection-switch re-assertion that follows the loop.The inner
catchstays exactly as it was — it is the per-server scrape failure the contract means by "swallows per-server failures". This adds the missing outercatchfor the cache layer, logging through the samedebugchannel.No behaviour change on the success or throttled paths;
inflightScrapecleanup was already in thefinallyand is untouched.Verified:
npx tsc --noEmitinapps/apiexits 0;npx vitest run src/modules/system/ test/modules/system/→ 13 files / 137 tests pass. Fullapps/apisuite is green apart from four pre-existing load-related 20s timeouts in unrelated files that also fail on unpatchedmain.No test here — the trigger is a cache-backend rejection rather than anything reachable through the public surface, and stubbing
cacheStoreto reject felt like more scaffolding than the nine-line fix warrants. Say the word and I'll add it.