perf(backend): run the full PGO solve only when new loop closures arrive#154
Open
harelb wants to merge 2 commits into
Open
perf(backend): run the full PGO solve only when new loop closures arrive#154harelb wants to merge 2 commits into
harelb wants to merge 2 commits into
Conversation
have_loopclosures_ is latched forever, so after the first LC every spin ran a full KimeraRpgo batch solve (1.86s x 314 spins for ONE LC in the box_7 run = 41% of backend time), starving the queue and lagging the published DSG by 15-20 min. Solve only when new LC factors arrived; between solves deform with the cached optimizer values (temp values cover not-yet-solved nodes).
backend/optimization and backend/mesh_update were never registered, so dsg_pgmo_status.csv optimize_time/mesh_update_time were always NaN.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BackendModule::stepgates optimization onhave_loopclosures_, which latches permanently once the first loop closure arrives:After the first LC, every backend spin for the rest of the session runs a full KimeraRpgo batch solve, even when no new factors arrived. Measured on a 38-min indoor bag (mit_infinite 2nd floor): one loop closure caused 314 full solves at ~1.9 s each (41% of total backend time); with a realistic LC stream the backend backlogs and the published DSG lags the live map by 15-20 minutes, never recovering.
Fix
have_new_loopclosures_is set (new LC factors actually arrived) or when forced.callUpdateFunctionsstill runs with the cached optimizer values (getTempValues()/getValues()), so new nodes/mesh keep deforming with the latest available correction; temp values cover not-yet-solved nodes. Once no LC has ever arrived, behavior is unchanged (emptyUpdateInfoas before).Since batch LM is order-independent and LC factors are added to the deformation graph on ingestion regardless, the final optimized map is unchanged — only redundant re-solves are eliminated.
Second commit:
logStatus()reads timersbackend/optimization/backend/mesh_update, which are never registered (the real timers aredsg_updater/optimizationandbackend/mesh_deformation), sodsg_pgmo_status.csv'soptimize_time/mesh_update_timecolumns were always NaN. Fixed the keys — this is also how the win above was measured.Measured results (fork, same bag)
Under a real 235-edge loop-closure stream, solves = LC batches exactly (56/56).
Notes