docs: add rollback procedure and ops runbook#2
Conversation
- rollback.md: git-revert as the primary rollback path (this repo is the source of truth, so fixing only in-cluster gets clobbered by ArgoCD's next sync), promote.yml re-roll to a known-good sha- tag as a second option, and a labeled break-glass kubectl/argocd rollback for when git/CI itself is down. - runbook.md: quick-reference table for common situations (new dev deploy, promote, bad version live, CI scan failures, ArgoCD OutOfSync/Degraded, adding an environment, bootstrap, secrets).
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces two new documentation files, docs/rollback.md and docs/runbook.md, which outline procedures for managing rollbacks and troubleshooting common operational scenarios in an ArgoCD-managed environment. The review feedback suggests clarifying the break-glass rollback instructions by advising operators to temporarily disable ArgoCD's auto-sync/self-heal to prevent immediate reversion of manual changes. Additionally, it recommends removing transient status information from the runbook to ensure the documentation remains evergreen.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| **This does not touch git.** ArgoCD will drift-detect and try to re-sync to | ||
| whatever `main` says on its next pass, which can silently undo your | ||
| break-glass fix. Treat this as a stopgap only — follow up immediately with | ||
| option 1 or 2 so git matches reality again. |
There was a problem hiding this comment.
If the ArgoCD Application has automated sync with selfHeal enabled, any manual kubectl rollout undo or argocd app rollback will be almost instantly reverted by ArgoCD's self-healing mechanism. To make this break-glass option effective, the runbook should instruct operators to temporarily disable auto-sync/self-heal (e.g., via the ArgoCD UI or CLI) before performing the manual rollback.
| **This does not touch git.** ArgoCD will drift-detect and try to re-sync to | |
| whatever `main` says on its next pass, which can silently undo your | |
| break-glass fix. Treat this as a stopgap only — follow up immediately with | |
| option 1 or 2 so git matches reality again. | |
| **This does not touch git.** If the ArgoCD application has automated sync with selfHeal enabled, ArgoCD will immediately detect the drift and re-sync to whatever main says, instantly undoing your manual rollback. | |
| To use this break-glass option: | |
| 1. Disable auto-sync/self-heal first (via the ArgoCD UI or argocd app set sample-app-<env> --sync-policy none). | |
| 2. Perform the manual rollback. | |
| 3. Treat this as a stopgap only — follow up immediately by committing the revert (Option 1 or 2) and re-enabling auto-sync. |
| | Need to deploy a new dev build | Merge to `main` on `sample-app` | CI builds, scans, pushes, and bumps the dev overlay automatically | | ||
| | Need to promote dev → staging or staging → prod | Run `promote.yml` (workflow_dispatch) on `sample-app` with the target env + `sha-xxxxxxx` tag | Requires the GitHub Environment approval for that env | | ||
| | A bad version is live in an environment | See `rollback.md` | Prefer `git revert` over direct kubectl | | ||
| | CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust `severity-cutoff` in `ci.yml` | Currently failing as of the last two runs — unresolved | |
There was a problem hiding this comment.
Hardcoding transient status like 'Currently failing as of the last two runs — unresolved' in a runbook can quickly become outdated and lead to confusion once the CI pipeline is fixed. It is better to keep the runbook evergreen by focusing on the troubleshooting steps and linking to an active issue or dashboard for the current status.
| | CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust `severity-cutoff` in `ci.yml` | Currently failing as of the last two runs — unresolved | | |
| | CI is failing on the "Scan image" step | Check the Anchore scan output in the Actions log for the CVE(s); either fix the base image / deps, or if it's an accepted risk, adjust severity-cutoff in ci.yml | Refer to active repository issues or security dashboards for any known/unresolved scan failures | |
1 participant
Adds two docs:
docs/rollback.md— git-revert as primary rollback path (repo is source of truth, in-cluster-only fixes get clobbered by next ArgoCD sync),promote.ymlre-roll to a known-good sha as second option, and a labeled break-glasskubectl/argocdprocedure for when git/CI is down.docs/runbook.md— quick-reference table for common ops situations (new dev deploy, promote, bad version live, CI scan failure, ArgoCD OutOfSync/Degraded, new environment, bootstrap, secrets).