[mariadb-replication-and-ha] Missing: Safe Galera bootstrap procedure after full cluster outage

[jana-dbaasnow]

Summary

The skill has no guidance on how to restart a Galera cluster after a
full outage. Bootstrapping the wrong node silently discards the most
recent writes committed before the outage. This is a data loss scenario
that agents currently have no way to prevent.

The problem

When all Galera nodes are down simultaneously (power outage, patch window,
cascading failure), the cluster cannot restart itself. A single node must
be bootstrapped as the new Primary Component, and the other nodes join
from it.

The critical rule: bootstrap the node with the highest
wsrep_last_committed sequence number. This is the node that has the
most recent committed transaction. Bootstrapping any other node means
the more recent transactions on other nodes are silently overwritten
during IST/SST.

Agents do not know this procedure and frequently suggest just starting
any node first, or restarting all nodes simultaneously.

What agents should know

# Step 1: On each node, recover the last committed GTID
# Do NOT start MariaDB normally — use recovery mode
mysqld --wsrep-recover 2>&1 | grep "Recovered position"
# Example output: Recovered position: 1e2c4b8a-...:1234567

# Step 2: Check grastate.dat on each node
cat /var/lib/mysql/grastate.dat
# The node with the highest seqno is the bootstrap candidate
# safe_to_bootstrap: 1 means this node was the last to leave cleanly

# Step 3: Bootstrap only the most advanced node
galera_new_cluster   # on the chosen node only

# Step 4: Start remaining nodes normally — they join from the bootstrap node
systemctl start mariadb   # on each remaining node

# WARNING: Never run galera_new_cluster on more than one node.
# It creates a new cluster, discarding the previous cluster's data.

Suggested addition

A cluster recovery subsection covering:

• Why bootstrap node selection matters for data integrity
• --wsrep-recover to identify the most advanced node
• grastate.dat and safe_to_bootstrap flag interpretation
• galera_new_cluster as the bootstrap command, with an explicit
  warning that it must only run on one node

---

Reviewed from an operations perspective by Jana Korapala, Founder and CEO,
DBaasNow. DBaasNow is a platform agnostic database lifecycle control plane
adding MariaDB to its catalog in Q3 2026.