feat(startup): hot startup — load hotStartupHours synchronously, fill retentionHours in background#1187
Open
efiten wants to merge 9 commits intoKpa-clawbot:masterfrom
Open
feat(startup): hot startup — load hotStartupHours synchronously, fill retentionHours in background#1187efiten wants to merge 9 commits intoKpa-clawbot:masterfrom
efiten wants to merge 9 commits intoKpa-clawbot:masterfrom
Conversation
…tStore wiring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When hotStartupHours > 0, Load() applies it as the initial time-window cutoff instead of retentionHours, enabling fast startup with a smaller in-memory set. Adds TestHotStartup_LoadsOnlyHotWindow and TestHotStartup_DisabledWhenZero to verify both code paths.
…nk loading with error recovery
…since/until predates oldestLoaded Also fix oldestLoaded to reflect the hot-window cutoff boundary (not the first packet's FirstSeen) so that the fallback threshold is accurate.
… in /api/perf/store
…e-append in loadChunk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1183
Summary
packetStore.hotStartupHoursconfig key (float64, default 0 = disabled). When set,Load()loads only that many hours of data synchronously, cutting cold startup from 5–20 minutes to under 60 seconds on large DBs.loadBackgroundChunks) fills the remainingretentionHourswindow in daily chunks after startup completes. Analytics indexes are rebuilt once at the end.QueryPacketsandQueryGroupedPacketscheckoldestLoadedand fall back todb.QueryPackets()for any query whoseSince/Untilpredates the in-memory window — covering days 8–30 permanently (beyondretentionHours) and the background-fill gap during startup./api/perf/storegainshotStartupHours,backgroundLoadComplete, andbackgroundLoadProgressfields so operators can monitor the fill.Configuration
hotStartupHours: 0(default) preserves existing behavior exactly.Test plan
TestHotStartupConfig_Clamp— clamping whenhotStartupHours > retentionHoursTestHotStartupConfig_ZeroIsDisabled— zero leaves feature disabledTestHotStartup_LoadsOnlyHotWindow— only hot-window packets in memory afterLoad()TestHotStartup_DisabledWhenZero— all retention packets loaded when disabledTestHotStartup_loadChunk_AddsOlderData— chunk merges correctly, ASC order maintainedTestHotStartup_BackgroundFillsToRetention— background goroutine fills toretentionHoursTestHotStartup_ChunkErrorRecovery— chunk SQL failure logged and skipped, loop terminatesTestHotStartup_SQLFallback_TriggeredForOldDate— query beforeoldestLoadedroutes to SQLTestHotStartup_SQLFallback_NotTriggeredForRecentDate— recent query stays in-memoryTestHotStartup_PerfStats— new fields present in/api/perf/storeresponse🤖 Generated with Claude Code