Skip to content

feat(startup): hot startup — load hotStartupHours synchronously, fill retentionHours in background#1187

Open
efiten wants to merge 9 commits intoKpa-clawbot:masterfrom
efiten:feat/hot-startup
Open

feat(startup): hot startup — load hotStartupHours synchronously, fill retentionHours in background#1187
efiten wants to merge 9 commits intoKpa-clawbot:masterfrom
efiten:feat/hot-startup

Conversation

@efiten
Copy link
Copy Markdown
Contributor

@efiten efiten commented May 10, 2026

Closes #1183

Summary

  • Adds packetStore.hotStartupHours config key (float64, default 0 = disabled). When set, Load() loads only that many hours of data synchronously, cutting cold startup from 5–20 minutes to under 60 seconds on large DBs.
  • A background goroutine (loadBackgroundChunks) fills the remaining retentionHours window in daily chunks after startup completes. Analytics indexes are rebuilt once at the end.
  • QueryPackets and QueryGroupedPackets check oldestLoaded and fall back to db.QueryPackets() for any query whose Since/Until predates the in-memory window — covering days 8–30 permanently (beyond retentionHours) and the background-fill gap during startup.
  • /api/perf/store gains hotStartupHours, backgroundLoadComplete, and backgroundLoadProgress fields so operators can monitor the fill.

Configuration

"packetStore": {
  "retentionHours": 168,
  "hotStartupHours": 24
}

hotStartupHours: 0 (default) preserves existing behavior exactly.

Test plan

  • TestHotStartupConfig_Clamp — clamping when hotStartupHours > retentionHours
  • TestHotStartupConfig_ZeroIsDisabled — zero leaves feature disabled
  • TestHotStartup_LoadsOnlyHotWindow — only hot-window packets in memory after Load()
  • TestHotStartup_DisabledWhenZero — all retention packets loaded when disabled
  • TestHotStartup_loadChunk_AddsOlderData — chunk merges correctly, ASC order maintained
  • TestHotStartup_BackgroundFillsToRetention — background goroutine fills to retentionHours
  • TestHotStartup_ChunkErrorRecovery — chunk SQL failure logged and skipped, loop terminates
  • TestHotStartup_SQLFallback_TriggeredForOldDate — query before oldestLoaded routes to SQL
  • TestHotStartup_SQLFallback_NotTriggeredForRecentDate — recent query stays in-memory
  • TestHotStartup_PerfStats — new fields present in /api/perf/store response

🤖 Generated with Claude Code

efiten and others added 9 commits May 9, 2026 20:47
…tStore wiring

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When hotStartupHours > 0, Load() applies it as the initial time-window
cutoff instead of retentionHours, enabling fast startup with a smaller
in-memory set. Adds TestHotStartup_LoadsOnlyHotWindow and
TestHotStartup_DisabledWhenZero to verify both code paths.
…since/until predates oldestLoaded

Also fix oldestLoaded to reflect the hot-window cutoff boundary (not the
first packet's FirstSeen) so that the fallback threshold is accurate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(startup): hot startup — load hot window only, background-fill to retentionHours, SQL fallback for older search

1 participant