diff --git a/docs/rollback.md b/docs/rollback.md new file mode 100644 index 0000000..031af57 --- /dev/null +++ b/docs/rollback.md @@ -0,0 +1,60 @@ +# Rollback + +This repo is the source of truth: what's in `main` is what ArgoCD applies. +That means rolling back is a **git operation**, not a `kubectl` operation — +if you fix it only in the cluster, ArgoCD will just re-apply the bad state on +its next sync. + +## 1. Preferred: revert the git commit + +Every deploy to an environment is one commit that bumps `newTag:` in that +overlay's `kustomization.yaml` (see `ci.yml` / `promote.yml` in `sample-app`). +So the fastest safe rollback is reverting that commit: + +```bash +# find the commit that bumped the bad tag in, e.g., prod +git log --oneline -- apps/sample-app/overlays/prod/kustomization.yaml + +# revert it +git revert +git push +``` + +ArgoCD picks up the revert on its next sync (dev/staging: automatic; +prod: automatic but not self-pruning — see README) and rolls the Deployment +back to the previous `sha-` tag. + +## 2. Roll forward to a known-good tag via `promote.yml` + +If you know the exact `sha-xxxxxxx` tag that was last good (check +`kustomization.yaml` history or ArgoCD's sync history for that app), you can +skip the revert and just re-promote it: + +1. GitHub → `sample-app` → Actions → **Promote** → Run workflow +2. `environment`: the broken one (`staging` or `prod`) +3. `image_tag`: the last known-good `sha-xxxxxxx` + +This runs `kustomize edit set image` and commits, same as a normal promotion, +so it stays auditable. + +## 3. Break-glass: direct kubectl/ArgoCD rollback + +Only if the above two are too slow for the incident (e.g. prod is down and +git/CI is unavailable): + +```bash +kubectl -n sample-app- rollout undo deployment/sample-app +# or: argocd app rollback sample-app- +``` + +**This does not touch git.** ArgoCD will drift-detect and try to re-sync to +whatever `main` says on its next pass, which can silently undo your +break-glass fix. Treat this as a stopgap only — follow up immediately with +option 1 or 2 so git matches reality again. + +## Finding "last known good" + +- `git log -p -- apps/sample-app/overlays//kustomization.yaml` — full + history of every tag that's ever been deployed to that environment. +- ArgoCD UI → app → History and Rollback — shows synced revisions with + timestamps, useful for correlating "when did this start breaking". diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..1597fa9 --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,16 @@ +# Runbook + +Quick lookup for common situations. If it's not here, check `rollback.md` +for anything deploy-related, or the ArgoCD UI for sync/health state. + +| Situation | What to do | Notes | +|---|---|---| +| Need to deploy a new dev build | Merge to `main` on `sample-app` | CI builds, scans, pushes, and bumps the dev overlay automatically | +| Need to promote dev → staging or staging → prod | Run `promote.yml` (workflow_dispatch) on `sample-app` with the target env + `sha-xxxxxxx` tag | Requires the GitHub Environment approval for that env | +| A bad version is live in an environment | See `rollback.md` | Prefer `git revert` over direct kubectl | +| CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust `severity-cutoff` in `ci.yml` | Currently failing as of the last two runs — unresolved | +| ArgoCD shows an app `OutOfSync` | Check what changed: `git diff` between the last synced revision and `main` for that overlay | Dev/staging auto-sync; prod auto-syncs too but won't prune deleted resources | +| ArgoCD shows an app `Degraded`/`Unhealthy` | `kubectl -n sample-app- get pods,events` first, then check the Deployment's image tag actually exists in GHCR | A pinned `sha-` tag that was never successfully pushed will manifest as `ImagePullBackOff` | +| Need to add/change a new environment | Copy an existing overlay under `apps/sample-app/overlays/`, add matching ArgoCD `Application` under `argocd/apps/`, add it to `argocd/app-of-apps.yaml` | Keep `namespace:` unique per env | +| Bootstrapping ArgoCD on a fresh cluster | Follow "Bootstrap ArgoCD" section in `README.md` | | +| Rotating/adding a secret the app needs | Not yet supported — no secrets management is wired up in this repo yet (tracked separately, likely External Secrets Operator + SSM Parameter Store) | Don't hand-roll a Secret manifest into git in the meantime |