fix(agent): prevent LVMVolumeGroup discover panic on empty lv_attr#308
Merged
Conversation
isThinPool indexed lv.LVAttr[0] without a length check, so any LV that lvs reports with an empty lv_attr (e.g. an LV observed mid create/delete) crashed the whole lvm-volume-group-discover-controller reconcile with "index out of range [0] with length 0", panic-looping the discoverer and stalling LVMVolumeGroup discovery on the node. Guard the index (mirroring utils.IsLVThinPool / monitoring.isThinPool) and reuse the helper in reconcileThinPoolsIfNeeded, which had the same unguarded access. Add a regression test covering the empty-attr case. Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
isThinPool(and the duplicated check inreconcileThinPoolsIfNeeded) indexedlv.LVAttr[0]without a length check. Whenlvsreports an LV with an emptylv_attr— which happens for an LV observed in a transient state (e.g. mid create/delete under concurrent LVM activity) — the access panics withindex out of range [0] with length 0.The panic occurs inside the
lvm-volume-group-discover-controllerreconcile (iterating the cached LV list). controller-runtime recovers it and immediately retries, so the discoverer panic-loops and stops updatingLVMVolumeGroupdiscovery/status for the node until the scan cache refreshes without the offending LV.Changes:
lvg.isThinPool: guard the index —len(lv.LVAttr) > 0 && lv.LVAttr[0] == 't'— mirroring the already-correctutils.IsLVThinPool/monitoring.isThinPool.reconcileThinPoolsIfNeeded: reuseisThinPool(lv)instead of the same unguardedstring(lv.LVAttr[0]) == "t".getThinPoolsskips an empty-lv_attrLV without panicking).Note:
llv.checkIfLVBelongsToLLVhas a similar unguardedlv.LVAttr[2]/[0]access reachable from one of its callers; left out of this focused fix but worth a follow-up.Why do we need it, and what problem does it solve?
Observed on a live cluster: the agent's discover controller panic-looped ~2900 times, which blocks
LVMVolumeGroupdiscovery and status reconciliation on the affected node for as long as the bad LV stays in the scan cache. A single transient LV state should not be able to take down the whole discoverer.What is the expected result?
The discover controller no longer panics when an LV reports an empty
lv_attr; such an LV is treated as "not a thin pool" and skipped, andLVMVolumeGroupdiscovery keeps working. The added unit test panics on the old code and passes on the fixed code (go test ./internal/controller/lvg/).Checklist