Skip to content

sample-app: PDB + RollingUpdate strategy to contain bad rollouts #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
David-Uka merged 1 commit into from
Jul 6, 2026
Merged

sample-app: PDB + RollingUpdate strategy to contain bad rollouts #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions apps/sample-app/base/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ spec:
selector:
matchLabels:
app: sample-app
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
Expand Down
1 change: 1 addition & 0 deletions apps/sample-app/base/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ resources:
- service.yaml
- hpa.yaml
- networkpolicy.yaml
- pdb.yaml
9 changes: 9 additions & 0 deletions apps/sample-app/base/pdb.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: sample-app
spec:
maxUnavailable: 1

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since the application uses a Horizontal Pod Autoscaler (hpa.yaml), the number of replicas can scale dynamically. Using a hardcoded absolute value of maxUnavailable: 1 can become highly restrictive as the application scales up (e.g., during node drains or cluster upgrades, only one pod can be evicted at a time, which significantly slows down cluster maintenance).

Consider using a percentage-based value like maxUnavailable: "33%" instead. This scales dynamically with the replica count while maintaining the same behavior for lower replica counts:

  • 1 replica (dev): 1 * 33% = 0.33 (rounded up to 1), allowing full disruption.
  • 2 replicas (staging): 2 * 33% = 0.66 (rounded up to 1), allowing at most 1 unavailable pod.
  • 3 replicas (prod): 3 * 33% = 0.99 (rounded up to 1), allowing at most 1 unavailable pod.
  maxUnavailable: "33%"

selector:
matchLabels:
app: sample-app
18 changes: 18 additions & 0 deletions docs/runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,21 @@ for anything deploy-related, or the ArgoCD UI for sync/health state.
| Need to add/change a new environment | Copy an existing overlay under `apps/sample-app/overlays/`, add matching ArgoCD `Application` under `argocd/apps/`, add it to `argocd/app-of-apps.yaml` | Keep `namespace:` unique per env |
| Bootstrapping ArgoCD on a fresh cluster | Follow "Bootstrap ArgoCD" section in `README.md` | |
| Rotating/adding a secret the app needs | Not yet supported — no secrets management is wired up in this repo yet (tracked separately, likely External Secrets Operator + SSM Parameter Store) | Don't hand-roll a Secret manifest into git in the meantime |

## Bad rollout containment

- **Rollout strategy**: `Deployment.spec.strategy` is `RollingUpdate` with
`maxUnavailable: 0, maxSurge: 1` — a bad rollout won't take down existing
healthy pods until the new ones pass readiness. Combined with the readiness
probe, a crash-looping new version stalls the rollout instead of replacing
the whole fleet.
- **PodDisruptionBudget**: `maxUnavailable: 1` on `sample-app` protects

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update the documentation to reflect the percentage-based maxUnavailable value if the suggestion to use "33%" is adopted.

Suggested change
- **PodDisruptionBudget**: `maxUnavailable: 1` on `sample-app` protects
- **PodDisruptionBudget**: `maxUnavailable: "33%"` on `sample-app` protects

against voluntary disruptions (node drains, cluster-autoscaler, manual
`kubectl delete`) stacking with an in-progress bad rollout and taking out
every pod at once. In `dev` (1 replica) this still allows full disruption
when the single pod is intentionally evicted — that's expected there.
- **Not yet wired**: the `PodCrashLooping` alert (in `cluster-observability`)
doesn't yet carry a `runbook_url` annotation pointing back here, and there's
no automated rollback (no Argo Rollouts analysis step) — today, detection
is automatic but response is still a human reading this doc. Out of scope
for this repo/PR; tracked separately.
Loading