Skip to content

fix(agent): prevent LVMVolumeGroup discover panic on empty lv_attr#308

Merged
astef merged 1 commit into
mainfrom
astef-fix-lvg-discover-empty-lvattr-panic
Jun 29, 2026
Merged

fix(agent): prevent LVMVolumeGroup discover panic on empty lv_attr#308
astef merged 1 commit into
mainfrom
astef-fix-lvg-discover-empty-lvattr-panic

Conversation

@astef

@astef astef commented Jun 26, 2026

Copy link
Copy Markdown
Member

Description

isThinPool (and the duplicated check in reconcileThinPoolsIfNeeded) indexed lv.LVAttr[0] without a length check. When lvs reports an LV with an empty lv_attr — which happens for an LV observed in a transient state (e.g. mid create/delete under concurrent LVM activity) — the access panics with index out of range [0] with length 0.

The panic occurs inside the lvm-volume-group-discover-controller reconcile (iterating the cached LV list). controller-runtime recovers it and immediately retries, so the discoverer panic-loops and stops updating LVMVolumeGroup discovery/status for the node until the scan cache refreshes without the offending LV.

Changes:

  • lvg.isThinPool: guard the index — len(lv.LVAttr) > 0 && lv.LVAttr[0] == 't' — mirroring the already-correct utils.IsLVThinPool / monitoring.isThinPool.
  • reconcileThinPoolsIfNeeded: reuse isThinPool(lv) instead of the same unguarded string(lv.LVAttr[0]) == "t".
  • Add a regression unit test (getThinPools skips an empty-lv_attr LV without panicking).

Note: llv.checkIfLVBelongsToLLV has a similar unguarded lv.LVAttr[2]/[0] access reachable from one of its callers; left out of this focused fix but worth a follow-up.

Why do we need it, and what problem does it solve?

Observed on a live cluster: the agent's discover controller panic-looped ~2900 times, which blocks LVMVolumeGroup discovery and status reconciliation on the affected node for as long as the bad LV stays in the scan cache. A single transient LV state should not be able to take down the whole discoverer.

panic: runtime error: index out of range [0] with length 0
  .../lvg.isThinPool(...)                                  utils.go:42
  .../lvg.getThinPools(...)                                discoverer.go:794
  .../lvg.(*Discoverer).GetLVMVolumeGroupCandidates(...)   discoverer.go:393

What is the expected result?

The discover controller no longer panics when an LV reports an empty lv_attr; such an LV is treated as "not a thin pool" and skipped, and LVMVolumeGroup discovery keeps working. The added unit test panics on the old code and passes on the fixed code (go test ./internal/controller/lvg/).

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

isThinPool indexed lv.LVAttr[0] without a length check, so any LV that
lvs reports with an empty lv_attr (e.g. an LV observed mid create/delete)
crashed the whole lvm-volume-group-discover-controller reconcile with
"index out of range [0] with length 0", panic-looping the discoverer and
stalling LVMVolumeGroup discovery on the node.

Guard the index (mirroring utils.IsLVThinPool / monitoring.isThinPool) and
reuse the helper in reconcileThinPoolsIfNeeded, which had the same
unguarded access. Add a regression test covering the empty-attr case.

Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
@astef astef self-assigned this Jun 29, 2026
@astef astef merged commit 42c74cc into main Jun 29, 2026
13 of 14 checks passed
@astef astef deleted the astef-fix-lvg-discover-empty-lvattr-panic branch June 29, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant