Skip to content

perf(backend): run the full PGO solve only when new loop closures arrive#154

Open
harelb wants to merge 2 commits into
MIT-SPARK:developfrom
harelb:perf/optimize-only-on-new-lcs
Open

perf(backend): run the full PGO solve only when new loop closures arrive#154
harelb wants to merge 2 commits into
MIT-SPARK:developfrom
harelb:perf/optimize-only-on-new-lcs

Conversation

@harelb

@harelb harelb commented Jul 8, 2026

Copy link
Copy Markdown

Problem

BackendModule::step gates optimization on have_loopclosures_, which latches permanently once the first loop closure arrives:

if ((config.optimize_on_lc && have_loopclosures_) || force_optimize_) {
    optimize(timestamp_ns);

After the first LC, every backend spin for the rest of the session runs a full KimeraRpgo batch solve, even when no new factors arrived. Measured on a 38-min indoor bag (mit_infinite 2nd floor): one loop closure caused 314 full solves at ~1.9 s each (41% of total backend time); with a realistic LC stream the backend backlogs and the published DSG lags the live map by 15-20 minutes, never recovering.

Fix

  • Solve only when have_new_loopclosures_ is set (new LC factors actually arrived) or when forced.
  • Between solves, callUpdateFunctions still runs with the cached optimizer values (getTempValues()/getValues()), so new nodes/mesh keep deforming with the latest available correction; temp values cover not-yet-solved nodes. Once no LC has ever arrived, behavior is unchanged (empty UpdateInfo as before).

Since batch LM is order-independent and LC factors are added to the deformation graph on ingestion regardless, the final optimized map is unchanged — only redundant re-solves are eliminated.

Second commit: logStatus() reads timers backend/optimization / backend/mesh_update, which are never registered (the real timers are dsg_updater/optimization and backend/mesh_deformation), so dsg_pgmo_status.csv's optimize_time/mesh_update_time columns were always NaN. Fixed the keys — this is also how the win above was measured.

Measured results (fork, same bag)

before after
solves per LC event ~314 (every spin, forever) exactly 1
backend spin mean 0.54 s 0.13-0.29 s
published-graph staleness after LC burst 15-20 min, permanent recovers when burst ends
objects-on-mesh median (map quality) 0.12 m 0.11 m

Under a real 235-edge loop-closure stream, solves = LC batches exactly (56/56).

Notes

harelb added 2 commits July 8, 2026 16:10
have_loopclosures_ is latched forever, so after the first LC every spin
ran a full KimeraRpgo batch solve (1.86s x 314 spins for ONE LC in the
box_7 run = 41% of backend time), starving the queue and lagging the
published DSG by 15-20 min. Solve only when new LC factors arrived;
between solves deform with the cached optimizer values (temp values
cover not-yet-solved nodes).
backend/optimization and backend/mesh_update were never registered, so
dsg_pgmo_status.csv optimize_time/mesh_update_time were always NaN.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant