Skip to content

fix(operator): lazy-load storage gate#7

Open
imlach wants to merge 2 commits into
mainfrom
fix/lazy-storage-gate
Open

fix(operator): lazy-load storage gate#7
imlach wants to merge 2 commits into
mainfrom
fix/lazy-storage-gate

Conversation

@imlach

@imlach imlach commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Summary

  • defer TrueNAS storage-gate construction until the drain/shutdown path reaches the storage-gate step
  • keep wake and already-converged reconciles from failing during backend construction when TrueNAS TLS is unavailable
  • add coverage that configured storage-gate errors surface at wait time instead of builder time

Live verification

  • All ElasticNode reconciles were failing with: backends for <node>: storage gate: truenas: dial wss://<storage-endpoint>/api/current ... x509: certificate has expired ... after 2026-06-29T22:18:30Z

Tests

  • go test ./...

@imlach

imlach commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

Review notes

The change does what it intends: RealBuilder runs on every reconcile via ClusterProvider.BackendsFor, so the eager TrueNAS dial made every ElasticNode reconcile (wake, converged no-ops) depend on TrueNAS availability. Deferring the dial into the StorageGateFunc closure scopes that dependency to the one drain-shutdown step that needs it. The dropped "storage gate: %w" wrapping is re-established (with the node name) in lifecycle.DrainShutdown, the deleted closeStorage error-path cleanups are genuinely obsolete, and the connection is torn down on both success and error paths.

Four findings, most severe first:

  1. The TrueNAS dial+login isn't bounded by the timeout the closure receivesinternal/operate/operate.go:196. truenas.New has no internal deadline (websocket dial + auth.login_ex are bounded only by ctx), and the reconcile context is cancel-only with no deadline. StorageTimeout is passed only to WaitDetached, so a TrueNAS endpoint that black-holes (dropped packets, rather than a fast TLS failure like the expired cert) hangs the storage-gate step — and the reconcile worker — indefinitely. The hazard predates this PR, but the fix is now one line inside the closure:

    ctx, cancel := context.WithTimeout(ctx, timeout)
    defer cancel()

    so dial + wait share the step's budget.

  2. Behavior change worth an explicit yes/no: a node missing iscsi_initiator_addr now gets cordoned and fully drained before the error surfaces, then sits Blocked retrying into the same wall until the inventory is fixed. Previously this pure config error failed at builder time, before evicting anything. Eager validation in RealBuilder would re-break wake for such nodes, but the identity check needs no network — a drain-path-only pre-validation (before cordon) could keep fail-fast for config errors while keeping the dial lazy. Related: a TrueNAS outage on a desiredPower=Off node now reports PhaseBlocked instead of PhaseError (retry/backoff behavior is identical; only the status label changed — relevant if anything alerts on Error).

  3. Three comments still describe the eager lifecycle: internal/operate/operate.go:27 ("TrueNAS wiring is read from the environment … at build time"), internal/operate/operate.go:51 (Backends.Close "e.g. the TrueNAS websocket"), internal/controller/elasticnode_controller.go:88 ("release the TrueNAS websocket etc. after this reconcile").

  4. TrueNAS env-presence check now lives in three places (lazyStorageGate, BuildStorageGate, cmd/nightwatch/drainshutdown.go:83) — a small trueNASConfigured() bool helper would keep them from drifting. Note the check in lazyStorageGate is load-bearing: the nil interface is what makes DrainShutdown record the step as "skipped" rather than a false success, so dedupe it, don't remove it.

Checked and fine: lazyStorageGate returns a true nil interface (skip logic intact); WaitDetached runs exactly once per drain op, so per-call dialing costs nothing extra; env can't realistically change between build and wait within the operator process; talos.New doesn't share this failure mode (it only parses config — gRPC dials lazily on first RPC).

Verdict: merge after the one-line timeout fix in (1) and the comment updates in (3); (2) is intentional per the new test but deserves a deliberate decision rather than riding in silently.

* fix(operator): bound lazy storage-gate dial by step timeout

Addresses review findings on #7's lazy storage-gate change
(#7 (comment)):

- lazyStorageGate's closure now wraps ctx in context.WithTimeout(timeout)
  before calling BuildStorageGate, so the unbounded TrueNAS dial+auth
  (truenas.New has no internal deadline) shares the step's timeout budget
  with WaitDetached instead of only bounding the wait. Guarded on
  timeout > 0 so a zero/unset StorageTimeout degrades to "unbounded,
  ctx-only" rather than an instantly-expired context, matching how other
  steps (e.g. waitPowerOff) treat <=0 as "no override".
- Reworded three comments left over from the eager storage-gate lifecycle
  (Config's env-at-build-time doc, Backends.Close's TrueNAS-websocket
  mention, and the reconciler's Close comment) to describe the current
  lazy dial/close-inside-the-gate-call behavior.
- Added operate.TrueNASConfigured(), deduplicating the three copies of the
  TrueNAS env-presence check (lazyStorageGate, BuildStorageGate, and the
  CLI's drain-shutdown plan display). lazyStorageGate and BuildStorageGate
  still return a nil gate when unconfigured; only the presence check moved.

The drain-path eager-identity-validation question raised in the same
review is intentionally not addressed here, pending a design decision.

* fix(operator): default storage-gate step timeout to 5 minutes

Review follow-ups on #8:

- lazyStorageGate now defaults timeout<=0 to 5 minutes (mirroring
  waitPowerOff) and always bounds ctx with context.WithTimeout, instead
  of guarding on timeout>0. The previous comment mis-cited waitPowerOff
  as treating <=0 as "no override" (it defaults to 5m), and leaving ctx
  unbounded for <=0 was incoherent anyway: iscsi.Gate.WaitClear applies
  WithTimeout unconditionally, so timeout=0 (reachable via the
  unvalidated --storage-timeout flag) meant "unbounded dial,
  instantly-expired wait". Now dial + wait share one coherent budget.
- drop the dead buildStorageGate CLI wrapper from
  cmd/nightwatch/drainshutdown.go (nothing calls it since the run path
  moved to operate.RealBuilder) and its unused lifecycle import; its
  no-creds test moves to the operate package, next to BuildStorageGate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant