feat(storage): add Ceph RBD storage gate (draft)#9
Draft
imlach wants to merge 1 commit into
Draft
Conversation
Add a second StorageGate backend alongside the TrueNAS iSCSI gate, for nodes whose backing storage is Ceph RBD (plan risk R5). Mirrors the iscsi/truenas split: internal/cephrbd is the backend-agnostic polling gate (WaitDetached, substring watcher match), internal/cephdash is a thin no-cgo HTTP client against the Ceph mgr dashboard API that feeds it raw per-image watcher text, robust to unconfirmed field naming the same way truenas.SessionTable is. Wiring in internal/operate: NIGHTWATCH_CEPH_* env (host/username/ password/images) + CephConfigured(), BuildCephStorageGate, and a new explicit inventory.NodeSpec.CephClientAddr identity field (no fallback to TalosEndpoint, matching StorageGateIdentity's philosophy). lazyStorageGate now picks TrueNAS vs Ceph by which env set is present, preserving the lazy-dial + nil-interface-when-unconfigured contract, and treats both being configured as a hard (loudly-failing) config error rather than guessing or gating on both. This is a draft: the Ceph dashboard endpoint/auth shape is not yet confirmed against a live cluster - see the PR description for open questions and alternatives considered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a second
lifecycle.StorageGatebackend for nodes whose data volumeslive on Ceph RBD rather than TrueNAS iSCSI, per the gap
internal/lifecycle'spackage doc explicitly calls out ("Ceph RBD watcher/map checks, NFS, etc. slot
in behind the same interface").
internal/cephrbd— the backend-agnostic polling gate, mirroringinternal/iscsiexactly: aWatcherListerseam,Gate.WaitDetached(ctx, clientAddr, timeout)polling until no watcher matches, and acase-insensitive substring
WatcherPresentmatch. Also includes an optionalCommandLister(shells out torbd status), mirroringiscsi.CommandListeras an alternative to the HTTP client below.
internal/cephdash— a thin, no-cgo HTTP client against the Ceph mgrdashboard REST API (
/api/authlogin,GET /api/block/image/<pool>/<image>per watched image), playing the role
internal/truenasplays for iSCSI.Like
truenas.SessionTable, it returns each response's raw body text,never a parsed struct, so the gate's substring match survives schema/field
drift.
internal/inventory: new explicitNodeSpec.CephClientAddrfield(
ceph_client_addr), the Ceph analogue ofISCSIInitiatorAddr— nofallback to
TalosEndpoint, same philosophy as the existingStorageGateIdentitydoc comment.internal/operate:NIGHTWATCH_CEPH_HOST/USERNAME/PASSWORD/IMAGESenv,CephConfigured(),CephStorageGateIdentity(),BuildCephStorageGate().lazyStorageGatenow picks TrueNAS vs Ceph by which env set is present,refactored through a shared
lazyGatehelper (keeps the just-landedstep-timeout-bounded-dial logic in one place for both backends). Both
configured at once is a loud runtime error, not a guess.
cmd/nightwatch/drainshutdown.go:--dry-runplan output now nameswhichever backend (if either) is configured, or flags the ambiguous-both
case.
Rebased onto the latest
fix/lazy-storage-gate(the step-timeout-bounding fixin #8 and the
buildStorageGatecleanup); no ceph-specific sim was added(optional per scope — see below).
Design decisions
Client approach (no cgo, so no go-ceph/librados): chose the Ceph mgr
dashboard REST API over the alternatives:
watcher state is read from the image header object directly (what
rbd statusdoes under the hood), which isn't one of the commands that moduleexposes. It wouldn't get us watcher data any more directly than dashboard.
rbd/cephCLI — no-cgo and simple, but needs a workingceph.conf+ keyring on whatever host runs nightwatch, which the truenasgate deliberately avoids (it talks to the target over the network instead
of needing local admin tooling). Kept as
cephrbd.CommandListeras anavailable alternative, unwired, for exactly the case where that's actually
the easier path in a given deployment.
local ceph.conf/keyring, plain HTTPS + username/password auth
(
ceph dashboard ac-user-create). Lowest friction to wire from a separateoperator host, same shape as the truenas websocket client (dial + login +
per-op calls + explicit close).
Important caveat, called out loudly in
internal/cephdash's package doc:this draft has not verified against a live cluster whether
GET /api/block/image/<pool>/<image>actually surfaces live watcher/mapperaddresses, or the exact path escaping the dashboard expects for the
<pool>/<image>spec. Because of that,WatcherTablereturns raw responsebytes rather than a parsed struct — if the field name, or even the whole
endpoint, turns out to be wrong, only
cephdash.WatcherTable's implementationneeds to change;
cephrbd.Gate's tested polling logic andoperate.go'swiring don't.
Identity field:
NodeSpec.CephClientAddr(ceph_client_addr), matched bycase-insensitive substring against the aggregated watcher text — same
contract as
iscsi_initiator_addr. No derived fallback: an address a nodehappens to be reachable on isn't necessarily the address Ceph's watcher
registration will show.
Which images to watch: Ceph has no cluster-wide "list every RBD watcher"
call — watchers are scoped per image.
NIGHTWATCH_CEPH_IMAGESis acomma-separated
pool/image[,pool/image...]list, checked (and requrednon-empty) only once
CephConfigured()is true, so an incomplete Ceph configfails loudly as a storage-gate error rather than silently looking
unconfigured. This is deliberately cluster-level env config, not per-node
inventory, matching how TrueNAS host/user/key are cluster-level too.
Backend selection when both are configured: hard error ("configure
exactly one storage-gate backend"), surfaced as a failed storage-gate
step (after cordon+drain have already run and recorded), not a
Backends-build-time error. Rationale: the storage gate is the harddata-safety barrier before power is pulled (R5) — guessing which backend an
ambiguous config meant, or gating on both sequentially, is worse than failing
loudly on the one step whose entire job is to fail loudly. Failing it after
cordon+drain (rather than before) means a config mistake still leaves the
node correctly cordoned/drained instead of untouched.
Laziness: preserved exactly —
lazyStorageGatestill returns a truenillifecycle.StorageGatewhen neither backend is configured (load-bearingfor the "skipped" step recording), and defers dialing either backend until
the storage-gate step actually runs, refactored through a
lazyGate(build, gateToken)helper shared by both backends so the just-landedstep-timeout-bounded-dial fix (#8) only needs to live in one place.
Ceph sim: skipped, per the task's "optional, skip unless trivial" — a
faithful sim would need to assume a specific dashboard response shape that
this draft hasn't confirmed, which felt like the overbuilt-guess this PR is
explicitly trying to avoid.
internal/cephdash's own tests usehttptestagainst a minimal fake dashboard instead.
Open questions (need Ross's input)
GET /api/block/image/<pool>/<image>actually return live watcheraddresses on your Ceph version? This is the load-bearing unknown. If not,
what does — a different dashboard endpoint, a custom mgr module, or should
this instead shell out via
cephrbd.CommandLister?/api/auth(what's wired), orwould a longer-lived dashboard API key / token be preferable operationally?
NIGHTWATCH_CEPH_IMAGESlistsufficient, or do you need per-node image sets (e.g. each node maps a
different boot volume), which would push this from cluster-level env into
per-node inventory config instead?
rather nightwatch gate on both backends sequentially in a mixed-storage
cluster?
ceph_client_addr(a bare IP) the right identity shape, or does yourcluster's watcher output include a port/nonce suffix that would make a
more precise (still-explicit) match worthwhile?
Tests run
gofmt -l .— cleango vet ./...— cleango build ./...— cleango test ./...— all passgo test -tags=integration ./...— all pass (existing TrueNAS/AMT simsunaffected; no Ceph sim added, see above)
go test -race ./...— all passThis is a draft: the shape (gate/client split, env wiring, backend
selection, laziness) is what's up for review, not a finished Ceph
integration — see open questions above before this touches a real cluster.