docs: add instructions for configuring cluster-autoscaler annotations

[Mallikarjunadevops]

Fixes #2291

What this PR does / why we need it:
This is a docs-only change that adds instructions to website/docs/install.md (and all past versioned docs) explaining how users can configure cluster-autoscaler.kubernetes.io/safe-to-evict: "true" for the audit pod using Helm. Since the audit pod uses an emptyDir mount for /tmp/audit, it can block Cluster Autoscaler node downscaling by default.

Rather than changing the default behavior out-of-the-box (which could surprise users upgrading), this PR ensures the capability is documented as an opt-in configuration for those who need it.

Which issue(s) this PR fixes:
Fixes #2291

Special notes for your reviewer:

• Updated the PR to be docs-only based on maintainer feedback to keep it opt-in.
• Refined the Helm instructions to use --set-string to prevent boolean type inference issues, and applied it strictly to the audit pod since the controller-manager no longer uses emptyDir.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Mallikarjunadevops / name: Mallikarjunadevops (1d141f3, 2d50043)

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the Kubernetes cluster-autoscaler safe-to-evict pod annotation to Gatekeeper-related workloads so the autoscaler is allowed to evict them during scale-down/optimization.

Changes:

• Add cluster-autoscaler.kubernetes.io/safe-to-evict: "true" to Gatekeeper audit-controller pod templates in rendered manifests.
• Add the same annotation to the Helm chart template for the audit deployment.
• Add the same annotation to the manager deployment manifest.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
manifest_staging/deploy/gatekeeper.yaml	Adds safe-to-evict annotation to the audit-controller PodTemplate metadata in staging manifest.
manifest_staging/charts/gatekeeper/templates/gatekeeper-audit-deployment.yaml	Hard-codes safe-to-evict annotation into the Helm template for the audit deployment.
deploy/gatekeeper.yaml	Adds safe-to-evict annotation to the audit-controller PodTemplate metadata in deploy manifest.
config/manager/manager.yaml	Adds safe-to-evict annotation to the manager PodTemplate metadata.

[JaydipGabani]

@Mallikarjunadevops why cant podAnnotations helm variable be used to solve the problem in the original issue?

[Mallikarjunadevops]

@JaydipGabani Good point! You are absolutely right that users can manually configure this via podAnnotations. However, since Gatekeeper's audit and manager deployments use emptyDir by default for /tmp, they silently block cluster downscaling out-of-the-box for users who might be unaware of this interaction.

To provide a better out-of-the-box experience while remaining configurable, I've updated this PR to remove the hardcoded annotations from the templates, and instead set them as defaults in the Helm values.yaml under both podAnnotations and auditPodAnnotations. This fully resolves the issue for Helm users by default while allowing them to override it if they prefer. Let me know if this approach looks better!

[JaydipGabani]

@Mallikarjunadevops I am hesitant to add default pod annotation, many users use upstream chart so this annotation will appear when they upgrade unless they manually chose to update. I would really like to keep this opted in so if there is a lack of documentation then we should update documentation for "how pod annotations can be set through helm" rather than changing defaults and surprise users after upgrading.

[Mallikarjunadevops]

@JaydipGabani Good call! You're absolutely right that changing the default behavior might surprise users who are not expecting it during an upgrade.

I have reverted the podAnnotations defaults in values.yaml back to {} so this remains opt-in as it was. Instead, I've added a section to the install.md documentation explaining exactly how users can easily pass these annotations through Helm if they are running the cluster-autoscaler.

Let me know if this new approach looks good to you!

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[JaydipGabani]

can you update the same in past versioned docs where this is true?

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[Mallikarjunadevops]

@JaydipGabani Just following up on this—I've gone ahead and thoroughly propagated this documentation update to all of the past versioned docs (website/versioned_docs/*/install.md) so that the information is available across the board.

Additionally, based on some automated PR review feedback, I've refined the instructions to be 100% accurate:

1. It now correctly targets only the audit pod (via auditPodAnnotations), since the controller-manager deployment does not actually use an emptyDir mount for /tmp.
2. The helm command snippet now utilizes --set-string instead of --set to prevent Helm from incorrectly inferring the annotation value as a boolean.
3. The PR description has been explicitly updated to reflect that this is a docs-only change.

Let me know if there's anything else you'd like me to tweak!