Skip to content

fix: say so when an upgraded cluster is serving an older CRD - #18

Merged
wrkode merged 1 commit into
mainfrom
fix/crd-upgrade-drop
Aug 9, 2026
Merged

fix: say so when an upgraded cluster is serving an older CRD#18
wrkode merged 1 commit into
mainfrom
fix/crd-upgrade-drop

Conversation

@wrkode

@wrkode wrkode commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Found while looking for release blockers: v0.1.1 would have shipped #10's rolling update silently broken for every upgraded release.

The trap

helm upgrade never updates a chart's crds/ — Helm installs those files once and never touches them. The apiserver then validates against the old schema and silently drops unknown fields. Not rejects: drops. The upgrade succeeds, the spec is stored without your fields, and everything reports success.

v0.1.1's CRD differs from v0.1.0's by exactly the two fields #10 added:

Field What an upgraded release would have done
spec.updatePolicy type: RollingUpdate accepted and discarded — pool reports Manual, never replaces a stale cell. The feature looks enabled and does nothing
status.cells[].templateHash the operator's writes vanish, so Updated compares against an empty hash — and an empty hash counts as current by design, so drift reports as up-to-date forever

The fix: make it loud

The manager embeds the CRD it was built against, compares property paths with the served schema at startup, and names what is being dropped:

ERROR  the cluster is serving an OLDER CRD than this operator was built against;
       the apiserver will SILENTLY DROP the fields below …
       {"missingFields":["v1alpha1.spec.updatePolicy","v1alpha1.status.cells[].templateHash"],
        "fix":"kubectl apply -f https://…/v0.1.1/config/crd/bases/cells.kubeswift.io_gpucellpools.yaml"}
  • make manifests syncs the embedded copy; CI diffs it, so the baseline cannot drift.
  • Unknown is never a verdict. An unreadable CRD reports CRD schema NOT compared with the missing grant named, at default verbosity — a check that quietly does not run is the same silent failure it exists to catch. (I hit exactly this while testing: the first run logged nothing because the live ClusterRole predated the grant and I had it at V(1).)
  • Non-fatal. Refusing to start would be a worse answer than naming the gap.
  • New grant: apiextensions.k8s.io/customresourcedefinitions: get, read-only.

installCRDs was a lie

It was referenced by nothing. The CRD ships in crds/, which Helm installs unconditionally, so installCRDs: false silently did nothing. Moving the CRD into templates/ to make the flag real would break upgrades of existing releases (Helm refuses to adopt a resource it does not own), so the knob is deleted and the manual step documented instead.

Cluster validation (dev)

Replaced the served CRD with v0.1.0's schema, restarted the manager → both fields named, correct fix URL. Restored it → CRD schema matches this build. Pool unaffected throughout.

That run found two defects in my own output, both fixed here: the fix command reconstructed the filename from the CRD's name (gpucellpools.cells.kubeswift.io.yaml, which 404s — controller-gen writes <group>_<plural>.yaml), and passing nil to Error() logged a stacktrace that read as a crash.

Also

  • docs/upgrading.md (new), linked from the docs index; NOTES.txt prints the command on upgrade only (.Release.IsUpgrade).
  • The release build stamps the version, so the printed URL points at that release rather than main.

make lint 0 issues, full suite green (8 new tests in internal/crdcheck), helm lint clean.

🤖 Generated with Claude Code

v0.1.1 adds two CRD fields, and `helm upgrade` never updates a chart's crds/. The
apiserver then silently drops what the old schema does not know: a pool asking for
`updatePolicy.type: RollingUpdate` would have been accepted and discarded — the
feature reporting Manual and never replacing a cell — and status.cells[].templateHash
writes would have vanished, so drift read as up-to-date forever (an empty hash counts
as current, deliberately). Nothing anywhere said a word.

The manager now embeds the CRD it was built against, compares property paths with
the served schema at startup, and names the exact fields being dropped plus the
kubectl command to fix it. `make manifests` syncs the embedded copy and CI diffs it,
so the baseline cannot drift. Unknown is never a verdict: an unreadable CRD reports
"not compared" with the missing grant named, at default verbosity — a check that
quietly does not run is the failure it exists to catch. Non-fatal throughout;
refusing to start would be a worse answer than naming the gap.

The `installCRDs` value is deleted. It was referenced by nothing: the CRD ships in
crds/, which Helm installs unconditionally, so setting it false silently did nothing.
Moving the CRD into templates/ to make the flag real would break upgrades of existing
releases (Helm refuses to adopt a resource it does not own), so the honest fix is to
remove the knob and document the manual step.

docs/upgrading.md is new, NOTES.txt prints the command on upgrade only, and the
release build stamps the version so the printed URL points at that release rather
than main.

Validated on the dev cluster by replacing the served CRD with v0.1.0's:

  missing: [v1alpha1.spec.updatePolicy, v1alpha1.status.cells[].templateHash]
  fix: kubectl apply -f https://.../v0.1.1/config/crd/bases/cells.kubeswift.io_gpucellpools.yaml

then restoring it: "CRD schema matches this build". Two output defects were found and
fixed by that run — the fix command reconstructed the filename from the CRD's name
and 404'd, and a nil error logged a stacktrace that read as a crash.

Signed-off-by: William Rizzo <william.rizzo@gmail.com>
@wrkode
wrkode merged commit cb9ce68 into main Aug 9, 2026
4 checks passed
@wrkode wrkode mentioned this pull request Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant