[PRODENG-3608] Fix inaccuracies and completeness gaps found during PRODENG-3608 e2e validation - #52
Open
sgajlekar-spec wants to merge 4 commits into
Conversation
…8 e2e validation Confirmed against actual ansible tasks/vars, terraform modules, and live cluster behavior during a full e2e run (provision, install, no-touch join, upgrade attempts): - install-bootc-mke3.md: mke_config_src must hold literal TOML content, not a path (ansible/tasks/mke-toml-config.yml uses copy:content, not src:); fix "last step" ordering claim for the SSH/sudo hardening play, which actually runs before the post-install controller installs. - install-controllers.md: same ordering fix; add a troubleshooting row for machine-config-controller's confirmed CrashLoopBackOff (chart 0.1.4 ships a ClusterRole with no upgrade.cattle.io/plans rule). - controller-security-analysis.md: flag that the RBAC table's machine-config- controller claims don't match what the pinned chart (0.1.4) actually ships, live-confirmed via the CrashLoopBackOff above. - provisioning.md: fix contradiction with the vSphere terraform module, which actually configures VMs via cloud-init (guestinfo), not kickstart. - provision-terraform-vsphere.md: document required vm_user/firmware vars (no defaults), fix the mngr/ctr naming example, note the terraform-version prerequisite gap vs the AWS doc. - provision-terraform-aws.md / terraform/aws/README.md: document that per-nodegroup user_data is silently ignored whenever is_bootc_based=true. - ansible-inventory-input.md: flag mke_url and the managers/workers group names as hard requirements, not just examples. - access-cluster.md: document the existing mke-client-bundle-playbook.yml automation as a third option. - machine-config-operations.md: add a troubleshooting row for the confirmed CrashLoopBackOff. - upgrade-with-controller.md: link the filed upstream issue (Mirantis/bootc-mirantis#178) for the overlayfs/overlay2 blocker, and state explicitly that the check is blind to the upgrade target (confirmed live against two separate ClusterUpgrade attempts); note backupDir has a CRD default. - upgrade-with-ansible.md: document upgrade_worker_batch_size. - mixed-cluster.md: add missing registry-access prerequisite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sgajlekar-spec
changed the title
nekwar
reviewed
Aug 10, 2026
| | `machine-config-controller` deployment not found in namespace `mke` | Its chart hardcodes `targetNamespace: system-upgrade`; look there instead | Not a fault — expected behavior | | ||
| | Locked out of SSH and sudo on every node | `disable_sshd_after_install`/`revoke_sudo_after_install` ran at the end of install (see [install runbook](install-bootc-mke3.md#post-install-automation)) | Break-glass recovery below | | ||
| | Locked out of SSH and sudo on every node | `disable_sshd_after_install`/`revoke_sudo_after_install` ran during install, before the controller installs (see [install runbook](install-bootc-mke3.md#post-install-automation)) | Break-glass recovery below | | ||
| | `machine-config-controller` pod stuck in `CrashLoopBackOff`, logs show `plans.upgrade.cattle.io is forbidden` and `Could not wait for Cache to sync` | Its pinned chart (`registry.mirantis.com/machine-config-controller/charts/machine-config-controller:0.1.4`, the `machine_config_controller_version` default in `vars/common-vars.yml`) ships a `ClusterRole` with no rule for `upgrade.cattle.io`/`plans`, so its `Plan`-watching informer never syncs and the manager exits every ~2 minutes | Known chart defect, not an installer misconfiguration — no workaround via Ansible vars. Track upstream fix in `machine-config-controller`'s chart RBAC template | |
Contributor
There was a problem hiding this comment.
instead of documenting it, I would suggest to fix it first without merging this PR
Collaborator
Author
There was a problem hiding this comment.
Agreed, reverted document to original state
nekwar
reviewed
Aug 10, 2026
| 2. IP addresses of managers, workers and gateway should be configured according to vSphere network configuration. DNS should be reachable by VMs. | ||
| 3. `cluster_name` variable will be used as a prefix for VM names. E.g. if cluster_name = "bootc-mke3-cluster", then names of managers will look like this: `bootc-mke3-cluster-ctr1`, `bootc-mke3-cluster-ctr2`, etc. Names of workers will look like this: `bootc-mke3-cluster-wrk1`, `bootc-mke3-cluster-wrk2`, etc. | ||
| 3. `cluster_name` variable will be used as a prefix for VM names. E.g. if cluster_name = "bootc-mke3-cluster", then names of managers will look like this: `bootc-mke3-cluster-mngr1`, `bootc-mke3-cluster-mngr2`, etc. Names of workers will look like this: `bootc-mke3-cluster-wrk1`, `bootc-mke3-cluster-wrk2`, etc. | ||
| 4. `vm_user` (required, no default) — the username the deployed VMs are |
Contributor
There was a problem hiding this comment.
this is not fully accurate. Yes, previously we had vsphere as a platform for our builds and it contained cloud-init with some other packages specific for vsphere. Now we're using bare variant to perform testing on vsphere with providing kickstart
Contributor
There was a problem hiding this comment.
that is probably my bad - I think I directed this comment.
Collaborator
Author
There was a problem hiding this comment.
Should I revert this changes? Are there any suggestions?
nekwar
reviewed
Aug 10, 2026
| > primarily used for internal testing. Where this documentation offers both | ||
| > mechanisms, prefer kickstart unless you are specifically working with a | ||
| > cloud build. | ||
| > is available on the cloud-platform builds (AMI/QCOW2) and is primarily |
Contributor
There was a problem hiding this comment.
Collaborator
Author
There was a problem hiding this comment.
Updated please check again
nekwar
reviewed
Aug 10, 2026
| > Use `-k`/`--insecure` only while MKE still serves its self-signed | ||
| > certificate; drop it once MKE has a trusted certificate installed. | ||
|
|
||
| ### Option 3 — Fetch the bundle with the provided Ansible playbook |
Contributor
There was a problem hiding this comment.
should we re-order options and make this the first one, as this is the simplest one?
nekwar
reviewed
Aug 10, 2026
| | Symptom | Likely cause | Remediation | | ||
| |---|---|---| | ||
| | `kubectl apply` fails `strict decoding error: unknown field "spec.reboot"` (or any other module) | The controller's CRD is stale relative to its chart | See the [controllers runbook](../installation-guide/install-controllers.md#machine-config-controllers-chart-is-the-exception--it-registry-pulls-by-default) for `kubectl apply -f <chart>/crds/*.yaml` | | ||
| | `machine-config-controller` never becomes healthy — `CrashLoopBackOff` before you've applied any `MachineConfigChange` at all | Known RBAC gap in chart `0.1.4` (the default `machine_config_controller_version`): its `ClusterRole` has no rule for `plans.upgrade.cattle.io`, so the controller can never sync and repeatedly exits | Not something this runbook covers — the controller isn't usable at all in this state. See the [controllers runbook](../installation-guide/install-controllers.md) troubleshooting table | |
Contributor
There was a problem hiding this comment.
nekwar
reviewed
Aug 10, 2026
|
|
||
| > [!WARNING] | ||
| > **Known issue, tracked separately**: the embedded `mirantis/ucp upgrade | ||
| > **Known issue, tracked upstream ([Mirantis/bootc-mirantis#178](https://github.com/Mirantis/bootc-mirantis/issues/178))**: the embedded `mirantis/ucp upgrade |
Contributor
There was a problem hiding this comment.
| > primarily used for internal testing. Where this documentation offers both | ||
| > mechanisms, prefer kickstart unless you are specifically working with a | ||
| > cloud build. | ||
| > is available on the cloud-platform builds (AMI/QCOW2) and is primarily |
Contributor
There was a problem hiding this comment.
we announce the qcow downloads in our public docs IIRC, so we should not consider it an internal artifact.
Collaborator
Author
There was a problem hiding this comment.
Changed, please check once again if its correct now
| ### 3.1 RBAC footprint | ||
|
|
||
| Source: kubebuilder markers in `internal/controller/machineconfigchange_controller.go:64-69`, rendered to `charts/machine-config-controller/templates/{clusterrole,role,leader-election-role}.yaml`. | ||
| Source: kubebuilder markers in `internal/controller/machineconfigchange_controller.go:64-69` at the analyzed commit `6b2b670`, rendered to `charts/machine-config-controller/templates/{clusterrole,role,leader-election-role}.yaml`. |
Contributor
There was a problem hiding this comment.
we should probably drop the specific commit reference, and keep this document either leading edge, or more abstract.
Collaborator
Author
There was a problem hiding this comment.
Agreed, removed commit version
| > `0.1.4` — the two were not confirmed to be the same artifact. Do not treat | ||
| > §3.1 as an accurate description of the default install's RBAC footprint | ||
| > until re-verified against the actual pinned chart version. See the | ||
| > [controllers runbook](../installation-guide/install-controllers.md) troubleshooting table for the live symptom. |
Contributor
There was a problem hiding this comment.
let's confirm that this isn't related to the issue of our current chart points to the url instead of an in-built chart: #51
…tistic-e2e-revalidate
PRODENG-3608-E2E-VALIDATION-REPORT.md and terraform/aws/inventory.yaml are local session artifacts, not meant to ship in this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While running a full e2e validation for PRODENG-3608 (provision → install → no-touch join → upgrade attempts), I cross-checked every doc under
docs/against the actual ansible tasks/vars, terraform modules, and live cluster behavior. This PR fixes what didn't hold up:install-bootc-mke3.md:mke_config_srcmust hold literal TOML content, not a path —ansible/tasks/mke-toml-config.ymlusescopy: content:, notsrc:. Also fixes the "last step" claim for the SSH/sudo hardening play, which actually runs before the post-install controller installs.install-controllers.md: same ordering fix, plus a troubleshooting row formachine-config-controller's confirmedCrashLoopBackOff(its pinned chart0.1.4ships aClusterRolewith noupgrade.cattle.io/plansrule).controller-security-analysis.md: flags that its RBAC table formachine-config-controllerdoesn't match what the pinned chart actually ships — live-confirmed via the CrashLoopBackOff above.provisioning.md: fixes a contradiction with the vSphere terraform module, which actually configures VMs via cloud-init (guestinfo), not kickstart, despite being ISO-provisioned.provision-terraform-vsphere.md: documents requiredvm_user/firmwarevars (no defaults), fixes themngr/ctrnaming example, notes a Terraform-version prerequisite gap vs. the AWS doc.provision-terraform-aws.md/terraform/aws/README.md: documents that per-nodegroupuser_datais silently ignored wheneveris_bootc_based=true.ansible-inventory-input.md: flagsmke_urland themanagers/workersgroup names as hard requirements, not just examples.access-cluster.md: documents the existingmke-client-bundle-playbook.ymlas a third, simpler option.machine-config-operations.md: adds a troubleshooting row for the confirmed CrashLoopBackOff.upgrade-with-controller.md: links the filed upstream issue (Mirantis/bootc-mirantis#178) for theoverlayfs/overlay2blocker, and states explicitly that the check is blind to the upgrade target — confirmed live against two separateClusterUpgradeattempts (same-version, and a real version bump to the latest release). Also notesbackupDirhas a CRD default.upgrade-with-ansible.md: documentsupgrade_worker_batch_size.mixed-cluster.md: adds a missing registry-access prerequisite.All fixes are backed by direct evidence gathered live this session (a real 11-node AWS e2e cluster) or by reading the actual source they describe (ansible tasks/vars, terraform variables/modules, live CRDs) — not just prose review.
Test plan
🤖 Generated with Claude Code