fix: a default chart install could not create a cell, or say why - #13
Merged
Conversation
Four defects, all found by installing the released chart on the dev cluster and driving a real pool through it rather than by reading the manifests. RBAC, secrets: the operator RENDERS a per-cell bootstrap Secret from the user's join template, but the marker granted get;list;watch. Every pool failed with "secrets is forbidden" on its first cell. This could not surface earlier because every prior end-to-end run used admin credentials. RBAC, events: the recorder writes through events.k8s.io, which nothing granted, so every event the operator emitted was rejected. The failure was visible only in the manager log — `kubectl describe gpucellpool` showed nothing at all. Chart image tag: appVersion carries no leading v while the release workflow pushes the tag verbatim, so the default tag resolved to 0.1.0 — a tag that was never published (v0.1.0 was). A default install ImagePullBackOff'd on its own image. ServiceMonitor: without honorLabels the pool's own `namespace` label was overwritten by the target's, landing as exported_namespace. Every alert would have named the operator's namespace instead of the pool's. CI grows three guards that fail on the pre-fix tree: the rendered default tag must equal v$(appVersion), and the rendered Role must carry the verbs the code actually uses. Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by installing the released v0.1.0 chart on the dev cluster and driving a real pool through it — not by reading manifests. All four are invisible to the test suite, which runs with admin credentials against a rendered-object harness.
secretsgrantedget;list;watchonlysecrets is forbiddenin the manager log, pool stuck with stale conditionsevents.k8s.iokubectl describe gpucellpoolshows no events at all; rejections only in the manager log0.1.0vs publishedv0.1.0ImagePullBackOffon a default installhonorLabelsnamespacelabel arrives asexported_namespace; every alert names the operator's namespaceThe operator renders a per-cell bootstrap Secret (
reconcileBootstrapSecret), socreate+updateare load-bearing.deleteis deliberately not granted — the per-cell Secret carries the pool's ownerRef and is garbage-collected.Validation
Live on the dev cluster, chart-installed operator, real GTX 1080 cell:
Reconciler error ... secrets is forbidden, no cell ever createdWorkloadClusterReachable=True→CapacityProviderReady=True→ cell created → GPU allocated → VM booted → workload Node joined and wentReadyhonorLabels: trueverified against dev's kube-prometheus-stack:namespacecomes back asgpu-cells, notexported_namespacelastError;GPUCellPoolWorkloadClusterUnreachablewentpendingfor a deliberately unreachable poolv0.1.0→ HTTP 200,0.1.0→ 404CI
Three new guards, each verified to fail on the pre-fix tree:
make lint0 issues,make testgreen.🤖 Generated with Claude Code