Manage Anti-Affinity Groups for the Control Plane

[bwagner5]

Currently, users of CAPOx can reference an existing anti-affinity group, but it would need to be created by something external (like a terraform config). Since anti-affinity groups are very light resources with a policy that support best-effort placement, CAPOx should create and manage an anti-affinity group for control plane instances.

OxideCluster can have an additional field:

spec.controlPlaneAntiAffinityPolicy: allow|fail (unset = disabled)

This would result in the OxideCluster Controller creating the anti-affintiy group with the specified policy and placing a reference in the status of the resource.

The Machine controller would:

1. Check if the machine is a control plane node
2. Retrieve the owning OxideCluster of the Machine and check the anti-affinity resource from the status.
3. Launch the CP instance in the anti-affinity group