Skip to content

docs: add rollback procedure and ops runbook #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
David-Uka merged 1 commit into from
Jul 6, 2026
Merged

docs: add rollback procedure and ops runbook #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/rollback.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Rollback

This repo is the source of truth: what's in `main` is what ArgoCD applies.
That means rolling back is a **git operation**, not a `kubectl` operation —
if you fix it only in the cluster, ArgoCD will just re-apply the bad state on
its next sync.

## 1. Preferred: revert the git commit

Every deploy to an environment is one commit that bumps `newTag:` in that
overlay's `kustomization.yaml` (see `ci.yml` / `promote.yml` in `sample-app`).
So the fastest safe rollback is reverting that commit:

```bash
# find the commit that bumped the bad tag in, e.g., prod
git log --oneline -- apps/sample-app/overlays/prod/kustomization.yaml

# revert it
git revert <bad-commit-sha>
git push
```

ArgoCD picks up the revert on its next sync (dev/staging: automatic;
prod: automatic but not self-pruning — see README) and rolls the Deployment
back to the previous `sha-` tag.

## 2. Roll forward to a known-good tag via `promote.yml`

If you know the exact `sha-xxxxxxx` tag that was last good (check
`kustomization.yaml` history or ArgoCD's sync history for that app), you can
skip the revert and just re-promote it:

1. GitHub → `sample-app` → Actions → **Promote** → Run workflow
2. `environment`: the broken one (`staging` or `prod`)
3. `image_tag`: the last known-good `sha-xxxxxxx`

This runs `kustomize edit set image` and commits, same as a normal promotion,
so it stays auditable.

## 3. Break-glass: direct kubectl/ArgoCD rollback

Only if the above two are too slow for the incident (e.g. prod is down and
git/CI is unavailable):

```bash
kubectl -n sample-app-<env> rollout undo deployment/sample-app
# or: argocd app rollback sample-app-<env> <history-id>
```

**This does not touch git.** ArgoCD will drift-detect and try to re-sync to
whatever `main` says on its next pass, which can silently undo your
break-glass fix. Treat this as a stopgap only — follow up immediately with
option 1 or 2 so git matches reality again.
Comment on lines +50 to +53

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the ArgoCD Application has automated sync with selfHeal enabled, any manual kubectl rollout undo or argocd app rollback will be almost instantly reverted by ArgoCD's self-healing mechanism. To make this break-glass option effective, the runbook should instruct operators to temporarily disable auto-sync/self-heal (e.g., via the ArgoCD UI or CLI) before performing the manual rollback.

Suggested change
**This does not touch git.** ArgoCD will drift-detect and try to re-sync to
whatever `main` says on its next pass, which can silently undo your
break-glass fix. Treat this as a stopgap only — follow up immediately with
option 1 or 2 so git matches reality again.
**This does not touch git.** If the ArgoCD application has automated sync with selfHeal enabled, ArgoCD will immediately detect the drift and re-sync to whatever main says, instantly undoing your manual rollback.
To use this break-glass option:
1. Disable auto-sync/self-heal first (via the ArgoCD UI or argocd app set sample-app-<env> --sync-policy none).
2. Perform the manual rollback.
3. Treat this as a stopgap only — follow up immediately by committing the revert (Option 1 or 2) and re-enabling auto-sync.


## Finding "last known good"

- `git log -p -- apps/sample-app/overlays/<env>/kustomization.yaml` — full
history of every tag that's ever been deployed to that environment.
- ArgoCD UI → app → History and Rollback — shows synced revisions with
timestamps, useful for correlating "when did this start breaking".
16 changes: 16 additions & 0 deletions docs/runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Runbook

Quick lookup for common situations. If it's not here, check `rollback.md`
for anything deploy-related, or the ArgoCD UI for sync/health state.

| Situation | What to do | Notes |
|---|---|---|
| Need to deploy a new dev build | Merge to `main` on `sample-app` | CI builds, scans, pushes, and bumps the dev overlay automatically |
| Need to promote dev → staging or staging → prod | Run `promote.yml` (workflow_dispatch) on `sample-app` with the target env + `sha-xxxxxxx` tag | Requires the GitHub Environment approval for that env |
| A bad version is live in an environment | See `rollback.md` | Prefer `git revert` over direct kubectl |
| CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust `severity-cutoff` in `ci.yml` | Currently failing as of the last two runs — unresolved |

@gemini-code-assist gemini-code-assist Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Hardcoding transient status like 'Currently failing as of the last two runs — unresolved' in a runbook can quickly become outdated and lead to confusion once the CI pipeline is fixed. It is better to keep the runbook evergreen by focusing on the troubleshooting steps and linking to an active issue or dashboard for the current status.

Suggested change
| CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust `severity-cutoff` in `ci.yml` | Currently failing as of the last two runs — unresolved |
| CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust severity-cutoff in ci.yml | Refer to active repository issues or security dashboards for any known/unresolved scan failures |

| ArgoCD shows an app `OutOfSync` | Check what changed: `git diff` between the last synced revision and `main` for that overlay | Dev/staging auto-sync; prod auto-syncs too but won't prune deleted resources |
| ArgoCD shows an app `Degraded`/`Unhealthy` | `kubectl -n sample-app-<env> get pods,events` first, then check the Deployment's image tag actually exists in GHCR | A pinned `sha-` tag that was never successfully pushed will manifest as `ImagePullBackOff` |
| Need to add/change a new environment | Copy an existing overlay under `apps/sample-app/overlays/`, add matching ArgoCD `Application` under `argocd/apps/`, add it to `argocd/app-of-apps.yaml` | Keep `namespace:` unique per env |
| Bootstrapping ArgoCD on a fresh cluster | Follow "Bootstrap ArgoCD" section in `README.md` | |
| Rotating/adding a secret the app needs | Not yet supported — no secrets management is wired up in this repo yet (tracked separately, likely External Secrets Operator + SSM Parameter Store) | Don't hand-roll a Secret manifest into git in the meantime |
Loading