Skip to content

Investigate coordinated simultaneous restart for shared NVMe subsystems #300

Description

@noctarius

Design investigation for #298 (tracking issue).

Goal

Determine whether and how to restart all pods owning all volumes on a shared NVMe subsystem simultaneously, so filesystem volumes restage cleanly after path recovery without causing cross-workload path loss. Output feeds the implementation sub-issue.

Questions to answer

  • Can a per-node DaemonSet Guardian coordinate restarts of pods that may span multiple nodes, or must a cluster-wide component (the operator) own this?
  • How to enumerate, cluster-wide, all volumes on a given subsystem and all pods owning them?
  • Ordering / atomicity: how to restart the set together (cordon/drain? simultaneous delete? a restage barrier?) so no pod reconnects to a half-torn-down subsystem.
  • Interaction with the opt-in/opt-out labels, PodDisruptionBudgets, and workload controllers (Deployment/StatefulSet/DaemonSet).
  • Partial-failure handling: what if some pods restart and others don't.
  • Blast-radius / safety limits and how this behaves at scale (high max volumes per subsystem).

Deliverable

A design proposal and a feasibility decision that the implementation sub-issue can build on.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions