Skip to content

Fix: Preserve webhook caBundle during failed Helm upgrades#224

Merged
briankane merged 3 commits intokubevela:mainfrom
roguepikachu:fix/admission-webhook-certs-on-helm-upgrades
Feb 9, 2026
Merged

Fix: Preserve webhook caBundle during failed Helm upgrades#224
briankane merged 3 commits intokubevela:mainfrom
roguepikachu:fix/admission-webhook-certs-on-helm-upgrades

Conversation

@roguepikachu
Copy link
Copy Markdown
Contributor

@roguepikachu roguepikachu commented Feb 9, 2026

Description of your changes

Prevent vela-workflow Helm installs and upgrades from leaving ValidatingWebhookConfiguration / MutatingWebhookConfiguration in a broken state with a placeholder caBundle: Cg==.

Problem

The Helm chart deploys webhook configurations with a hardcoded placeholder caBundle: Cg== and relies on a post-install/post-upgrade Job to patch the real CA bundle. If Helm fails before the patch Job completes (timeout, image pull error, controller CrashLoopBackOff with --wait, etc.), the webhooks persist with an invalid caBundle — silently bypassing admission validation.

Fix (mirrors kubevela/kubevela#6919)

  1. Preserve existing caBundle with Helm lookup — on upgrade, the webhook templates read the existing resource from the cluster and reuse its caBundle instead of overwriting with Cg==. Falls back to Cg== only on fresh install (expected; the patch Job handles it).

  2. Add post-rollback hook — the patch Job and its RBAC resources now also run after Helm rollbacks to restore the caBundle.


Checklist

  • Read and followed KubeVela's contribution process
  • Related Docs updated properly (for new features or configuration options)
  • Run make reviewable to ensure this PR is ready for review
  • Added backport release-x.y labels to auto-backport this PR if necessary

How has this code been tested

Reproduced on a k3d cluster (K3s v1.31.5, arm64):

Step Action Result
1 Installed baseline with kubevela/vela-workflow:0.6.0 Confirmed real caBundle (LS0tLS1..., 756 bytes) and failurePolicy: Fail
2 Simulated failing upgrade by breaking the patch Job's --secret-name Post-upgrade hook times out
3 Before fix Both webhook configs reverted to caBundle: Cg== (4 bytes) ❌
4 After fix Both webhook configs preserved the real caBundle (756 bytes) ✅

Reference: kubevela/kubevela#6919 (same fix applied to the vela-core chart)


Special notes for your reviewer

Consideration Details
Helm version Requires Helm 3.2+ for lookup function support
Dry-run compatibility helm template / --dry-run still works — lookup returns nothing without cluster access, templates fall back to Cg==, and the patch Job handles it at runtime
Cert-manager path Unaffected — when admissionWebhooks.certManager.enabled=true, patch jobs are disabled and cert-manager injects the caBundle via annotations

Summary by cubic

Preserves the admission webhooks’ caBundle on Helm installs, upgrades, and rollbacks to avoid broken webhooks when hooks fail. Requires Helm 3.2+.

  • Bug Fixes
    • Use Helm lookup to reuse the existing caBundle on upgrade; default to Cg== on fresh installs.
    • Add post-rollback hook for the patch job and its RBAC to restore caBundle after rollbacks.
    • Add a CI workflow that simulates a failed upgrade and verifies the caBundle is preserved.

Written for commit 53d7669. Summary will update on new commits.

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.19%. Comparing base (d7db9c4) to head (53d7669).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #224      +/-   ##
==========================================
+ Coverage   62.49%   70.19%   +7.70%     
==========================================
  Files          62       62              
  Lines        4415     5009     +594     
==========================================
+ Hits         2759     3516     +757     
+ Misses       1324     1142     -182     
- Partials      332      351      +19     
Flag Coverage Δ
e2etests 21.39% <ø> (?)
unit-test 64.22% <ø> (+1.73%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@roguepikachu roguepikachu force-pushed the fix/admission-webhook-certs-on-helm-upgrades branch from 1c4f105 to 2a46685 Compare February 9, 2026 06:26
@roguepikachu roguepikachu changed the title Fix/admission webhook certs on helm upgrades Fix: Preserve webhook caBundle during failed Helm upgrades Feb 9, 2026
@roguepikachu roguepikachu force-pushed the fix/admission-webhook-certs-on-helm-upgrades branch from 2a46685 to 54d711a Compare February 9, 2026 06:30
…ssion webhooks

---
Signed-off-by: Anaswara Suresh <anaswarasuresh2212@gmail.com>
Co-Authored-By: Anaswara Suresh <anaswarasuresh2212@gmail.com>

Signed-off-by: Ayush Kumar <ayushshyamkumar888@gmail.com>
@roguepikachu roguepikachu force-pushed the fix/admission-webhook-certs-on-helm-upgrades branch from 54d711a to 98e5e6f Compare February 9, 2026 06:30
@vishal210893
Copy link
Copy Markdown
Contributor

https://github.com/kubevela/kubevela/blob/master/.github/workflows/webhook-upgrade-validation.yml

Similar to the above action, create a verification GitHub action for this fix

roguepikachu and others added 2 commits February 9, 2026 13:55
Signed-off-by: Anaswara Suresh <anaswarasuresh2212@gmail.com>
Signed-off-by: Ayush Kumar <ayushshyamkumar888@gmail.com>
@roguepikachu
Copy link
Copy Markdown
Contributor Author

https://github.com/kubevela/kubevela/blob/master/.github/workflows/webhook-upgrade-validation.yml

Similar to the above action, create a verification GitHub action for this fix

@vishal210893 added the GHA, here is a sample run from my fork

@briankane briankane merged commit b4f001b into kubevela:main Feb 9, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants