Skip to content

feat(storage): add Ceph RBD storage gate (draft)#9

Draft
imlach wants to merge 1 commit into
fix/lazy-storage-gatefrom
feat/ceph-storage-gate
Draft

feat(storage): add Ceph RBD storage gate (draft)#9
imlach wants to merge 1 commit into
fix/lazy-storage-gatefrom
feat/ceph-storage-gate

Conversation

@imlach

@imlach imlach commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a second lifecycle.StorageGate backend for nodes whose data volumes
live on Ceph RBD rather than TrueNAS iSCSI, per the gap internal/lifecycle's
package doc explicitly calls out ("Ceph RBD watcher/map checks, NFS, etc. slot
in behind the same interface").

  • internal/cephrbd — the backend-agnostic polling gate, mirroring
    internal/iscsi exactly: a WatcherLister seam, Gate.WaitDetached(ctx, clientAddr, timeout) polling until no watcher matches, and a
    case-insensitive substring WatcherPresent match. Also includes an optional
    CommandLister (shells out to rbd status), mirroring iscsi.CommandLister
    as an alternative to the HTTP client below.
  • internal/cephdash — a thin, no-cgo HTTP client against the Ceph mgr
    dashboard REST API (/api/auth login, GET /api/block/image/<pool>/<image>
    per watched image), playing the role internal/truenas plays for iSCSI.
    Like truenas.SessionTable, it returns each response's raw body text,
    never a parsed struct, so the gate's substring match survives schema/field
    drift.
  • internal/inventory: new explicit NodeSpec.CephClientAddr field
    (ceph_client_addr), the Ceph analogue of ISCSIInitiatorAddr — no
    fallback to TalosEndpoint, same philosophy as the existing
    StorageGateIdentity doc comment.
  • internal/operate: NIGHTWATCH_CEPH_HOST/USERNAME/PASSWORD/IMAGES env,
    CephConfigured(), CephStorageGateIdentity(), BuildCephStorageGate().
    lazyStorageGate now picks TrueNAS vs Ceph by which env set is present,
    refactored through a shared lazyGate helper (keeps the just-landed
    step-timeout-bounded-dial logic in one place for both backends). Both
    configured at once is a loud runtime error, not a guess.
  • cmd/nightwatch/drainshutdown.go: --dry-run plan output now names
    whichever backend (if either) is configured, or flags the ambiguous-both
    case.

Rebased onto the latest fix/lazy-storage-gate (the step-timeout-bounding fix
in #8 and the buildStorageGate cleanup); no ceph-specific sim was added
(optional per scope — see below).

Design decisions

Client approach (no cgo, so no go-ceph/librados): chose the Ceph mgr
dashboard REST API over the alternatives:

  • mgr/restful module — maps arbitrary mon/mgr commands to HTTP, but RBD
    watcher state is read from the image header object directly (what rbd status does under the hood), which isn't one of the commands that module
    exposes. It wouldn't get us watcher data any more directly than dashboard.
  • shell out to rbd/ceph CLI — no-cgo and simple, but needs a working
    ceph.conf + keyring on whatever host runs nightwatch, which the truenas
    gate deliberately avoids (it talks to the target over the network instead
    of needing local admin tooling). Kept as cephrbd.CommandLister as an
    available alternative, unwired, for exactly the case where that's actually
    the easier path in a given deployment.
  • dashboard API (chosen) — ships enabled-by-default since Nautilus, no
    local ceph.conf/keyring, plain HTTPS + username/password auth
    (ceph dashboard ac-user-create). Lowest friction to wire from a separate
    operator host, same shape as the truenas websocket client (dial + login +
    per-op calls + explicit close).

Important caveat, called out loudly in internal/cephdash's package doc:
this draft has not verified against a live cluster whether
GET /api/block/image/<pool>/<image> actually surfaces live watcher/mapper
addresses, or the exact path escaping the dashboard expects for the
<pool>/<image> spec. Because of that, WatcherTable returns raw response
bytes rather than a parsed struct — if the field name, or even the whole
endpoint, turns out to be wrong, only cephdash.WatcherTable's implementation
needs to change; cephrbd.Gate's tested polling logic and operate.go's
wiring don't.

Identity field: NodeSpec.CephClientAddr (ceph_client_addr), matched by
case-insensitive substring against the aggregated watcher text — same
contract as iscsi_initiator_addr. No derived fallback: an address a node
happens to be reachable on isn't necessarily the address Ceph's watcher
registration will show.

Which images to watch: Ceph has no cluster-wide "list every RBD watcher"
call — watchers are scoped per image. NIGHTWATCH_CEPH_IMAGES is a
comma-separated pool/image[,pool/image...] list, checked (and requred
non-empty) only once CephConfigured() is true, so an incomplete Ceph config
fails loudly as a storage-gate error rather than silently looking
unconfigured. This is deliberately cluster-level env config, not per-node
inventory, matching how TrueNAS host/user/key are cluster-level too.

Backend selection when both are configured: hard error ("configure
exactly one storage-gate backend"), surfaced as a failed storage-gate
step
(after cordon+drain have already run and recorded), not a
Backends-build-time error. Rationale: the storage gate is the hard
data-safety barrier before power is pulled (R5) — guessing which backend an
ambiguous config meant, or gating on both sequentially, is worse than failing
loudly on the one step whose entire job is to fail loudly. Failing it after
cordon+drain (rather than before) means a config mistake still leaves the
node correctly cordoned/drained instead of untouched.

Laziness: preserved exactly — lazyStorageGate still returns a true
nil lifecycle.StorageGate when neither backend is configured (load-bearing
for the "skipped" step recording), and defers dialing either backend until
the storage-gate step actually runs, refactored through a lazyGate(build, gateToken) helper shared by both backends so the just-landed
step-timeout-bounded-dial fix (#8) only needs to live in one place.

Ceph sim: skipped, per the task's "optional, skip unless trivial" — a
faithful sim would need to assume a specific dashboard response shape that
this draft hasn't confirmed, which felt like the overbuilt-guess this PR is
explicitly trying to avoid. internal/cephdash's own tests use httptest
against a minimal fake dashboard instead.

Open questions (need Ross's input)

  1. Does GET /api/block/image/<pool>/<image> actually return live watcher
    addresses on your Ceph version?
    This is the load-bearing unknown. If not,
    what does — a different dashboard endpoint, a custom mgr module, or should
    this instead shell out via cephrbd.CommandLister?
  2. Auth mode: username/password against /api/auth (what's wired), or
    would a longer-lived dashboard API key / token be preferable operationally?
  3. Which images per cluster: is a flat NIGHTWATCH_CEPH_IMAGES list
    sufficient, or do you need per-node image sets (e.g. each node maps a
    different boot volume), which would push this from cluster-level env into
    per-node inventory config instead?
  4. Both backends configured: agree with hard-erroring, or would you
    rather nightwatch gate on both backends sequentially in a mixed-storage
    cluster?
  5. Is ceph_client_addr (a bare IP) the right identity shape, or does your
    cluster's watcher output include a port/nonce suffix that would make a
    more precise (still-explicit) match worthwhile?

Tests run

  • gofmt -l . — clean
  • go vet ./... — clean
  • go build ./... — clean
  • go test ./... — all pass
  • go test -tags=integration ./... — all pass (existing TrueNAS/AMT sims
    unaffected; no Ceph sim added, see above)
  • go test -race ./... — all pass

This is a draft: the shape (gate/client split, env wiring, backend
selection, laziness) is what's up for review, not a finished Ceph
integration — see open questions above before this touches a real cluster.

Add a second StorageGate backend alongside the TrueNAS iSCSI gate, for
nodes whose backing storage is Ceph RBD (plan risk R5). Mirrors the
iscsi/truenas split: internal/cephrbd is the backend-agnostic polling
gate (WaitDetached, substring watcher match), internal/cephdash is a
thin no-cgo HTTP client against the Ceph mgr dashboard API that feeds
it raw per-image watcher text, robust to unconfirmed field naming the
same way truenas.SessionTable is.

Wiring in internal/operate: NIGHTWATCH_CEPH_* env (host/username/
password/images) + CephConfigured(), BuildCephStorageGate, and a new
explicit inventory.NodeSpec.CephClientAddr identity field (no fallback
to TalosEndpoint, matching StorageGateIdentity's philosophy).
lazyStorageGate now picks TrueNAS vs Ceph by which env set is present,
preserving the lazy-dial + nil-interface-when-unconfigured contract,
and treats both being configured as a hard (loudly-failing) config
error rather than guessing or gating on both.

This is a draft: the Ceph dashboard endpoint/auth shape is not yet
confirmed against a live cluster - see the PR description for open
questions and alternatives considered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant