RFC: NVMe-oF orchestrator coexistence — nvme-discoverd, ownership registry, and exclusion list

[martin-belanger]

As NVMe-oF deployments grow, hosts increasingly run multiple tools that manage NVMe-oF connections — nvme-stas, dracut/initramfs scripts, and soon a new daemon nvme-discoverd. Without coordination these tools can conflict: one orchestrator may disconnect a controller another is actively managing, or connect to a controller another has deliberately excluded.

This discussion covers a proposed set of four RFCs that address orchestrator coexistence:

• rfc-nvme-orchestrator-coexistence — frames the problem, defines the orchestrator hierarchy, shows how the pieces fit together
• rfc-nvme-registry — a lightweight ownership registry in /run/nvme/registry/ so orchestrators can declare ownership and disconnect-all avoids stepping on live connections (implementation PR: libnvmf: NVMe Controller Ownership Registry (V2) #3425)
• rfc-nvme-exclusion — a human-administered exclusion list at /etc/nvme/exclusions/ to prevent orchestrators from auto-connecting to controllers the admin wants excluded
• rfc-nvme-discoverd — a new connect-only daemon that manages NBFT boot controllers, FC-discovered controllers, and statically configured controllers, using systemd transient units for per-controller lifecycle management

The full documents are in PR #3442 (DO-NOT-MERGE) where you can leave inline comments on specific sections.

High-level feedback here is very welcome — especially on whether the overall architecture fits how you see these tools being used in practice.

[martin-belanger]

cc: @igaw @mwilck @tbzatek @MichaelRabek — would appreciate your feedback on the proposed architecture, either here for high-level discussion or as inline comments on PR #3442.

[tbzatek]

+ Cc: @johnmeneghini

[martin-belanger]

Review feedback incorporated — updated RFCs coming

First, thank you all for the thorough reviews. I've worked through every comment, and the four RFCs have been revised to reflect them. I'll post the updated versions shortly; this note summarizes what changed, with the biggest item first.

The headline: the discoverd configuration file has been redesigned

This is the most substantial change. The original discoverd.conf was modeled on nvme-stas's format — a flat list of controller = entries, each carrying its full set of connection parameters inline. It could express everything a connection needs; what it lacked was any notion of grouping. There was no way to capture that a set of controllers belongs to the same host, or that several paths belong to the same subsystem, so every controller = line had to independently and precisely reproduce the shared information — the host NQN, the subsystem NQN, the TLS/auth keys, and so on. That redundancy is error-prone (the repeated fields must be kept consistent across every line by hand) and it makes the file hard to read: you cannot see at a glance which controllers sit under which host, or which paths serve the same subsystem. The redesign introduces that grouping, in a human-navigable INI form — mirroring the host → subsystem → port structure of libnvme's config.json, but without the JSON/json-c dependency and with comments and self-documenting templates.

The new model is split across two locations with distinct roles:

• discoverd.conf — the daemon's own configuration and the default connection parameters, in three sections: [Global] (discovery-source toggles + global defaults that apply to every connection), and [Discovery Controller Defaults] / [I/O Controller Defaults] (per-type defaults — e.g. kato = 30 for a DC vs 5 for an IOC — which are also the configuration point for auto-discovered controllers that have no section of their own).
• connections.d/*.conf — the actual connections, one file per host persona: an optional [Host] identity block (omit it to connect as the system default identity), followed by [Discovery Controller] and [Subsystem] endpoint sections. The role is the section name; there is no type= key.

Within an endpoint section, each controller = line is one path to that endpoint; repeating it is how multipath is expressed (any count ≥ 1). The address syntax is the same one nvme-stas uses and similar to the nvme connect command-line options — controller = transport=tcp; traddr=…; trsvcid=…; host-iface=… — which keeps host-traddr/host-iface available per path (they genuinely differ per path in multi-NIC multipath) and works uniformly across transports.

Parameters resolve most-specific-first: controller = line > endpoint section > [Host] > type defaults > [Global] > kernel default. Per-link security (the TLS PSK / DH-CHAP key) sits at the subsystem tier, because the PSK is bound to the (hostnqn, subsysnqn) pair and is therefore constant across all paths to that subsystem.

Conceptually, discoverd is a connection manager: its runtime unit is the controller, a connection is a controller, and a [Subsystem] with N paths simply expands into N I/O-controller connections at parse time. The config files resolve into two static lists — the configured Discovery Controllers and the configured I/O Controllers; IOCs learned at runtime from a Discovery Log Page or AEN are discovered, not configured, and are out of scope for the file format.

A nice side effect: because the controller = address syntax matches nvme-stas, the two daemons' configuration formats converge — I'm considering moving nvme-stas onto this same model, so an operator who learns one learns the other.

Other changes from the review

• NBFT and the exclusion list. I relaxed the "protect NBFT at all cost" rule: the exclusion list now applies to NBFT controllers too — it is the administrator's explicit, root-only instruction, and is the supported way to take a boot path out of service for testing or maintenance. owner=nbft still protects boot devices from other orchestrators via the registry; only discoverd's own reconnect yields to an explicit exclusion. The NBFT unavailable SSNS flag is now treated as the point-in-time hint the Boot Spec says it is (attempt anyway and retry, rather than give up). And root-backing controllers get no ExecStop= at shutdown — the kernel holds the connection until the block device is closed, so we don't drive a teardown into a still-mounted root.
• Registry. Documented the seqnum attribute (the cleanup-race tiebreaker), and kept the registry's normative attribute set deliberately minimal (owner + seqnum) — the contract's value is in staying small and unambiguous.
• Legacy autoconnect cleanup. The inventory was tightened: the inert dracut snippet is removed outright (as Fedora's build already does), the manual rd.nvmf boot path is acknowledged as a third initramfs mechanism, and the NBFT-specific running-system components are slated for removal.
• Coexistence. Updated to reflect that libblockdev and UDisks are moving toward registry participation (an optional owner argument; UDisks supplying owner='udisks' and actively respecting exclusions).

Still settling

Two items are intentionally left open pending more input rather than guessed at:

• What UDisks intends for future mDNS browsing — a third independent mDNS browser on a host (on top of nvme-discoverd and nvme-stas) raises a "who owns mDNS browsing" coordination question I'd rather settle before multiple layers do it independently.

I'll post the revised RFCs shortly. Thanks again for the careful reading — it materially improved the design.