Skip to content

Fix TUI lag: debounce Hera pane kick + gate tick's Hera rebuild on change-detection - #924

Merged
anutron merged 1 commit into
masterfrom
argus/fix-tui-lag-and-kick-storm
Aug 4, 2026
Merged

Fix TUI lag: debounce Hera pane kick + gate tick's Hera rebuild on change-detection#924
anutron merged 1 commit into
masterfrom
argus/fix-tui-lag-and-kick-storm

Conversation

@anutron

@anutron anutron commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes two distinct, confirmed causes of severe Argus TUI lag/memory pressure (Aaron's dogfooded TUI observed at ~59GB RSS / 46% RAM / 36% CPU after ~14-15h uptime, while the daemon (280MB) and session-supervisor (24MB) stayed small and normal — a client-side TUI problem, not daemon/supervisor):

  1. Kick storm on ordinary rail navigation. Hera's own layout swings a bound pane's width between full-terminal-width (fullscreen/Details mode) and roughly half-width (split mode) purely from moving the rail cursor between a coordinator row and a worker row — no resize or fullscreen toggle needed. This alone crosses agent.ShouldKickRerender's RerenderMargin (15 cols), so a fast Cmd+Arrow traversal across several rows binds and kicks (Session.Stop() + --session-id restart + full conversation replay) every transiently-bound task in quick succession.

    Fix: HeraPage.maybeKickPaneRerender now debounces the kick behind a 300ms wall-clock dwell (hera.KickDebounce). The first Draw that finds a newly-bound task past the margin arms a pending kick instead of firing immediately; only a later Draw, once the dwell elapses AND the same task is still bound, actually kicks. A rebind to a different task before the dwell elapses discards the old pending kick un-fired. No new goroutine — checked on the same Draw cadence, mirroring hera.Refresher's existing goroutine-free style. The anti-corruption kick mechanism itself (agent.ShouldKickRerender, RerenderMargin, idle/prompt/pending gates, exit-handler resume) is completely unchanged — only when it fires moved by ~300ms.

  2. Unconditional O(total-history) work on the 1s UI tick. App.onTickrefreshTasksWithIDs fetches every task row every second; while the Hera tab is active it also drives HeraPage.doRefreshBuildModel (fetches ALL orchestrators including fully-archived history, all live+latest bindings, re-fetches the full task list a second time) → Rail.buildRows() (runs canonicalParents()/structuralReach() graph algorithms over the WHOLE model, archived included) — every single tick, regardless of whether anything changed. This scales with total historical role/orchestrator/binding count (~900+ for Aaron), not active-agent count, matching the reported "stalling while sitting still with only 5-6 active agents."

    Fix: doRefresh now gates the BuildModel+SetModel rebuild behind cheap change-detection (HeraPage.shouldRebuild/markRebuilt): a SQLite PRAGMA data_version fingerprint (near-O(1), catches any daemon/other-process-driven DB write) combined with maps.Equal checks on the four per-tick runtime maps also fed into the model (needsInput/sessionIdle/sessionRunning/sustainedActive — bounded by active session count, so cheap even with 900+ historical roles). Either changing triggers a rebuild, so the rail's spinner/needs-input glyphs never freeze while agents are active even when the DB itself is quiet. PRAGMA data_version has a documented same-connection blind spot (a write through the SAME connection doesn't bump what that connection reads back); closed by HeraPage.Refresh() itself (not just the App's heraRefresh wrapper — see design.md Decision 5 for why the narrower first attempt broke an existing test) invalidating the gate before flushing, so its own "forces an immediate rebuild" contract holds regardless of caller. Also de-dupes a redundant second Tasks() fetch BuildModel was performing every tick via a small reader-wrapper (tasksReader), avoiding a BuildModel signature change that would've touched ~42 unrelated test call sites.

Measurement (honest, not assumed)

internal/tui/hera/doRefresh_bench_test.go (go test -bench, permanent regression-guard benchmarks, never run during plain go test ./...) seeded ~900 roles/~900 bindings across 450 archived orchestrators + 1 active one, matching Aaron's reported scale:

per-tick cost
Before (unconditional rebuild every tick) 34.3ms, 29.7MB, 132,903 allocs
After (steady-state, idle, nothing changed) 844ns, 400B, 12 allocs

~35,000x reduction in per-tick allocation/CPU cost while idle — at 1 tick/sec this eliminates ~30MB/s of allocation churn, strong evidence of a major contributor to sustained GC/heap pressure.

What this does NOT prove: allocation-rate reduction is not the same as retained-memory proof — Go's GC reclaims transient garbage, so this benchmark alone cannot confirm what fraction (if any) of the reported 59GB RSS was genuinely retained-forever vs. GC-pressure-inflated heap targets vs. a separate, still-unidentified leak. Confirming that requires an actual longitudinal RSS comparison of the real TUI dogfooded for a comparable multi-hour period — not performed in this PR (infeasible within a single PR's time budget) — flagged as a named follow-up in design.md's Open Questions.

Explicitly out of scope (named follow-ups, not silently dropped)

  • Widening the change-detection gate to refreshTasksWithIDs's own base reads (db.Tasks(), both ListMetaByNamespace calls, ManagedTaskIDs()) — more un-audited local-write call sites than the Hera-mutation chokepoint this gate relies on.
  • Lazy-loading archived orchestrators/roles inside Rail.buildRows()'s graph algorithms so an actually-needed rebuild is also structurally cheaper, not just less frequent — deferred given the risk to rail.go's heavily invariant-laden fold/nesting logic (long bug history documented in gotchas/hera-view.md).
  • Not deployed to the live dogfood daemon/TUI — that's a separate step for after review.

Full design rationale (7 numbered decisions, risks/trade-offs, migration plan, open questions) in openspec/changes/archive/2026-08-04-fix-hera-tick-and-kick-perf/design.md, archived in this same PR per repo convention.

Test plan

  • New/updated unit tests for the kick debounce (internal/tui/hera/panes_test.go): fast multi-row traversal never kicks, genuine dwell-and-stay still kicks once, unbind-mid-dwell-then-rebind-same-task re-arms correctly (caught a real bug via TDD — see design.md Decision 2).
  • New unit tests for the change-detection gate (internal/tui/hera/changegate_test.go): first-call-always-true, quiescent-skips, DB-fingerprint-change triggers, each of the 4 runtime maps individually triggers, unsupported-fingerprint/remote-mode always rebuilds, InvalidateChangeGate, Refresh() forces rebuild despite the same-connection blind spot, doRefresh integration, tasks-dedup via a call-counting reader wrapper.
  • New DB.DataVersion test (internal/db/db_test.go): cross-connection change visible, same-connection blind spot confirmed, stable with no writes — using two real file-backed connections (t.TempDir(), never touches real ~/.argus/).
  • make pre-pr (build/vet/fmt-check/lint-pr) clean. vuln fails only on 3 pre-existing stdlib CVEs (CI continue-on-error, confirmed unrelated to this diff). test-cover-gate green at 88.8% (floor 88%) once two pre-existing, documented environmental issues are excluded: this hera-worker sandbox's own ARGUS_* env vars leaking into 2 internal/agent profile-env tests (passes with them unset; CI has none), and TestSmoke_NewTaskFormPaste's documented pre-existing -race flake (confirmed flaky even in isolation, unrelated to any file this PR touches). A full clean run with the sandbox env excluded passed fully green.
  • Full internal/tui, internal/tui/hera, internal/db suites pass with no regressions.

🤖 Generated with Claude Code

…tection

Fixes two distinct causes of TUI lag/memory pressure Aaron hit dogfooding
(~59GB RSS after ~14-15h, daemon/supervisor both normal — client-side only):

1. Ordinary rail navigation between a coordinator row and a worker row alone
   swings a bound pane's width past agent.ShouldKickRerender's margin (no
   resize/fullscreen toggle needed), so a fast Cmd+Arrow traversal kicked
   (Session.Stop()+restart+full replay) every transiently-bound task in
   quick succession. maybeKickPaneRerender now debounces the kick behind a
   300ms wall-clock dwell, arm-then-fire per pane, goroutine-free.

2. The 1s UI tick's Hera rail rebuild (BuildModel + Rail.buildRows()) ran
   unconditionally every tick regardless of activity, doing O(total historical
   role/orchestrator/binding count) work — archived history included — even
   while idle. doRefresh now gates the rebuild on cheap change-detection: a
   SQLite PRAGMA data_version fingerprint (catches daemon-driven changes)
   combined with equality checks on the four per-tick runtime maps (catches
   agent-activity-driven changes the DB alone wouldn't show). Measured at
   ~900-role scale: 34ms/30MB/133k allocs per rebuild before, ~850ns/400B/12
   allocs per tick when idle after — a ~35,000x steady-state reduction. Also
   de-dupes a redundant second Tasks() fetch BuildModel was doing per tick.

This reduces per-tick cost dramatically and is strong evidence toward the
sustained RSS growth, but does not by itself prove it accounts for the full
59GB (allocation-rate reduction isn't the same as retained-memory proof) —
flagged as a named follow-up requiring an actual dogfood RSS comparison.

Both fixes are scoped and documented in
openspec/changes/archive/2026-08-04-fix-hera-tick-and-kick-perf/ (proposal,
design with decisions/risks/open-questions, delta specs, tasks). Explicitly
out of scope, named follow-ups: widening the gate to the plain task list's
own base reads, and lazy-loading archived rail data structurally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Merging this branch changes the coverage (1 decrease, 2 increase)

Impacted Packages Coverage Δ 🤖
github.com/drn/argus/internal/db 89.53% (-0.02%) 👎
github.com/drn/argus/internal/tui 83.00% (+0.03%) 👍
github.com/drn/argus/internal/tui/hera 92.71% (+0.68%) 👍

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/drn/argus/internal/db/db.go 77.61% (+0.56%) 67 (+6) 52 (+5) 15 (+1) 👍
github.com/drn/argus/internal/tui/app.go 77.96% (+0.12%) 2622 (+1) 2044 (+4) 578 (-3) 👍
github.com/drn/argus/internal/tui/hera/page.go 91.03% (+4.75%) 390 (+33) 355 (+47) 35 (-14) 👍
github.com/drn/argus/internal/tui/hera/panes.go 94.07% (-0.28%) 135 (+11) 127 (+10) 8 (+1) 👎
github.com/drn/argus/internal/tui/heraactions.go 76.06% (ø) 518 394 124

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/drn/argus/internal/db/db_test.go
  • github.com/drn/argus/internal/tui/hera/changegate_test.go
  • github.com/drn/argus/internal/tui/hera/doRefresh_bench_test.go
  • github.com/drn/argus/internal/tui/hera/panes_test.go

@anutron
anutron merged commit 0039606 into master Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant