fix: say so when an upgraded cluster is serving an older CRD - #18
Merged
Conversation
v0.1.1 adds two CRD fields, and `helm upgrade` never updates a chart's crds/. The apiserver then silently drops what the old schema does not know: a pool asking for `updatePolicy.type: RollingUpdate` would have been accepted and discarded — the feature reporting Manual and never replacing a cell — and status.cells[].templateHash writes would have vanished, so drift read as up-to-date forever (an empty hash counts as current, deliberately). Nothing anywhere said a word. The manager now embeds the CRD it was built against, compares property paths with the served schema at startup, and names the exact fields being dropped plus the kubectl command to fix it. `make manifests` syncs the embedded copy and CI diffs it, so the baseline cannot drift. Unknown is never a verdict: an unreadable CRD reports "not compared" with the missing grant named, at default verbosity — a check that quietly does not run is the failure it exists to catch. Non-fatal throughout; refusing to start would be a worse answer than naming the gap. The `installCRDs` value is deleted. It was referenced by nothing: the CRD ships in crds/, which Helm installs unconditionally, so setting it false silently did nothing. Moving the CRD into templates/ to make the flag real would break upgrades of existing releases (Helm refuses to adopt a resource it does not own), so the honest fix is to remove the knob and document the manual step. docs/upgrading.md is new, NOTES.txt prints the command on upgrade only, and the release build stamps the version so the printed URL points at that release rather than main. Validated on the dev cluster by replacing the served CRD with v0.1.0's: missing: [v1alpha1.spec.updatePolicy, v1alpha1.status.cells[].templateHash] fix: kubectl apply -f https://.../v0.1.1/config/crd/bases/cells.kubeswift.io_gpucellpools.yaml then restoring it: "CRD schema matches this build". Two output defects were found and fixed by that run — the fix command reconstructed the filename from the CRD's name and 404'd, and a nil error logged a stacktrace that read as a crash. Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while looking for release blockers: v0.1.1 would have shipped #10's rolling update silently broken for every upgraded release.
The trap
helm upgradenever updates a chart'scrds/— Helm installs those files once and never touches them. The apiserver then validates against the old schema and silently drops unknown fields. Not rejects: drops. The upgrade succeeds, the spec is stored without your fields, and everything reports success.v0.1.1's CRD differs from v0.1.0's by exactly the two fields #10 added:
spec.updatePolicytype: RollingUpdateaccepted and discarded — pool reportsManual, never replaces a stale cell. The feature looks enabled and does nothingstatus.cells[].templateHashUpdatedcompares against an empty hash — and an empty hash counts as current by design, so drift reports as up-to-date foreverThe fix: make it loud
The manager embeds the CRD it was built against, compares property paths with the served schema at startup, and names what is being dropped:
make manifestssyncs the embedded copy; CI diffs it, so the baseline cannot drift.CRD schema NOT comparedwith the missing grant named, at default verbosity — a check that quietly does not run is the same silent failure it exists to catch. (I hit exactly this while testing: the first run logged nothing because the live ClusterRole predated the grant and I had it atV(1).)apiextensions.k8s.io/customresourcedefinitions: get, read-only.installCRDswas a lieIt was referenced by nothing. The CRD ships in
crds/, which Helm installs unconditionally, soinstallCRDs: falsesilently did nothing. Moving the CRD intotemplates/to make the flag real would break upgrades of existing releases (Helm refuses to adopt a resource it does not own), so the knob is deleted and the manual step documented instead.Cluster validation (dev)
Replaced the served CRD with v0.1.0's schema, restarted the manager → both fields named, correct fix URL. Restored it →
CRD schema matches this build. Pool unaffected throughout.That run found two defects in my own output, both fixed here: the fix command reconstructed the filename from the CRD's name (
gpucellpools.cells.kubeswift.io.yaml, which 404s — controller-gen writes<group>_<plural>.yaml), and passingniltoError()logged a stacktrace that read as a crash.Also
docs/upgrading.md(new), linked from the docs index;NOTES.txtprints the command on upgrade only (.Release.IsUpgrade).main.make lint0 issues, full suite green (8 new tests ininternal/crdcheck),helm lintclean.🤖 Generated with Claude Code