densify-dev · kubexautomation · Jun 26, 2026
@@ -52,21 +52,41 @@ spec:
 
 ## Pause Controls
 
-Use the `rightsizing.kubex.ai/pause-until` annotation to temporarily or permanently block automation for a pod, a supported workload owner, or an entire namespace.
+Use pause annotations to temporarily or permanently block automation.
 
-Supported values:
+Pod-scoped whole-pod pause:
+
+- `rightsizing.kubex.ai/pause-until: <RFC3339|infinite>`
+- `rightsizing.kubex.ai/pause-reason: <string>`
+
+Container-scoped skip:
+
+- `rightsizing.kubex.ai/skip-containers: "app,sidecar"`
+
+Supported values for every `pause-until` key:
 
 - RFC3339 timestamp
 - `infinite`
 
-Example:
+Pod example:
 
 ```yaml
 spec:
   template:
     metadata:
       annotations:
         rightsizing.kubex.ai/pause-until: "2026-04-01T00:00:00Z"
+        rightsizing.kubex.ai/pause-reason: "maintenance window"
+```
+
+Container example:
+
+```yaml
+spec:
+  template:
+    metadata:
+      annotations:
+        rightsizing.kubex.ai/skip-containers: "app,sidecar"
 ```
 
 Namespace example:
@@ -83,21 +103,29 @@ metadata:
 
 Behavior:
 
-- new pods inherit `rightsizing.kubex.ai/pause-until` and `rightsizing.kubex.ai/pause-reason` from supported workload owners during admission
-- existing owned pods are reconciled to inherit the same pause annotations from the workload owner
-- pods in a paused namespace are skipped even when the pod itself has no pause annotation
+- pod-level `rightsizing.kubex.ai/pause-until` blocks whole pod and appears in `rightsizing summary.failedChecks` as `pause-active`
+- `rightsizing.kubex.ai/skip-containers` prunes only matching container actions and appears in `rightsizing summary.appliedFilters` as `container-skip-active`
+- sibling container actions still proceed when only one container is skipped
+- pod annotation key beats owner annotation; no merge across pod and owners
+- empty pod `skip-containers` value means skip none and disables owner fallback for that pod
+- nearest supported owner with non-empty annotation wins when pod key is absent; no merge across multiple owners
+- empty owner `skip-containers` values are ignored
+- new pods do not inherit `skip-containers`; it is resolved directly from pod or owner at evaluation time
+- existing owned pods are not reconciled to copy `skip-containers`
+- pods in paused namespace are skipped even when pod itself has no pause annotation
 - namespace pause annotations are evaluated at runtime only and are not copied onto pods
-- when both pod and namespace pauses are active, the pod-level pause reason wins
-- webhook mutation skips paused pods
-- controller-side proactive execution skips paused pods
+- namespace pause supports only pod-scoped keys
+- webhook mutation skips whole pod only for pod-level or namespace-level pauses; `skip-containers` still allows non-skipped sibling mutations
+- controller-side proactive execution skips whole pod only for pod-level or namespace-level pauses; `skip-containers` filters matching actions only
 - time-based pauses automatically resume after expiration
 
 Notes:
 
 - pod-local pause annotations are still supported
-- namespace-local pause annotations use the same `rightsizing.kubex.ai/pause-reason` message format in `rightsizing summary` failed checks
-- when a pause annotation was inherited from the workload owner, the controller tracks that internal inheritance state so it can safely remove the inherited value later
-- if a pause annotation exists only on the pod and was not inherited, the controller treats it as pod-local and does not remove it during workload reconciliation
+- `skip-containers` can be set on pod or supported workload owner, but is not propagated or stored as inherited control state
+- namespace-local pause annotations use same `rightsizing.kubex.ai/pause-reason` message format in `rightsizing summary` failed checks
+- when pause annotation was inherited from workload owner, controller tracks internal inheritance state so it can safely remove inherited value later
+- if pause annotation exists only on pod and was not inherited, controller treats it as pod-local and does not remove it during workload reconciliation
 
 ## Safety Controls
 

@@ -62,7 +62,7 @@ Use the [Tuning Guide](./Tuning-Guide.md) for leader election tolerance, webhook
 
 ### What prevents unsafe resizes?
 
-The controller filters or blocks actions based on webhook health, protected namespaces, pause annotations, HPA/VPA detection, quota and LimitRange checks, node headroom, and workload readiness checks. Pause annotations can be set directly on a pod or on a supported workload owner, in which case new pods inherit them during admission and existing owned pods are reconciled to the same pause state.
+The controller filters or blocks actions based on webhook health, protected namespaces, pause annotations, HPA/VPA detection, quota and LimitRange checks, node headroom, and workload readiness checks. Pod-level `rightsizing.kubex.ai/pause-until` blocks whole pod. `rightsizing.kubex.ai/skip-containers` removes only matching container actions and still allows sibling container actions to proceed. Pod annotation key beats owner annotation. If pod key is present with an empty value, that pod skips no containers and does not fall back to owners. If pod key is absent, nearest supported owner with non-empty value wins. Values are never merged. Namespace pause supports pod-level keys only.
 
 For the complete safety model and where each control is configured, see [Safety Controls](./Safety-Controls.md).
 
@@ -76,6 +76,8 @@ Not by default. Protected namespace patterns exclude well-known platform namespa
 
 Check `GlobalConfiguration`, policy objects, events, and `rightsizing summary` logs in that order. If new pods are being created but expected mutation is missing, also review the webhook fail-open notes in the [Tuning Guide](./Tuning-Guide.md#admission-webhook-fail-open-semantics).
 
+For multi-container pods, also check whether only some actions were filtered. `failedChecks` means whole pod blocked. `appliedFilters` with `container-skip-active`, `hpa-resource-managed`, or similar means only specific container/resource actions were removed.
+
 ### Why is a matching policy not enough?
 
 Because the controller still needs a valid recommendation and a plan that survives all strategy and safety filters.
@@ -37,7 +37,7 @@ How to interpret `retry` in the context of this document:
 | --- | --- | --- | --- | --- | --- |
 | `pod-terminating` | controller pod pre-check | always on (controller path) | Blocks if pod has `deletionTimestamp` | no retry from check itself | `pod is terminating` |
 | `namespace-protected` | pre-check | `GlobalConfiguration.spec.protectedNamespacePatterns` | Blocks if pod namespace matches a protected namespace pattern | no retry from check itself | `namespace "<namespace>" is protected` |
-| `pause-active` | pre-check | `spec.safetyChecks.enablePauseUntilAnnotationCheck` | Blocks when `rightsizing.kubex.ai/pause-until` is active (`infinite` or future RFC3339 time) on the pod or on the pod namespace. New pods can inherit pause annotations from supported workload owners, and existing owned pods are reconciled to the same inherited pause state. Namespace pause annotations are evaluated at runtime only and are not copied onto pods. Pod-local pause annotations remain valid and take precedence over namespace pauses. | no retry from check itself | `automation paused by annotation` or `automation paused: <pause-reason>` |
+| `pause-active` | pre-check | `spec.safetyChecks.enablePauseUntilAnnotationCheck` | Blocks whole pod when pod-scoped `rightsizing.kubex.ai/pause-until` is active (`infinite` or future RFC3339 time) on pod or pod namespace. New pods can inherit pod-scoped pause annotations from supported workload owners, and existing owned pods are reconciled to same inherited pause state. Namespace pause annotations are evaluated at runtime only and are not copied onto pods. Pod-local pause annotations remain valid and take precedence over namespace pauses. | no retry from check itself | `automation paused by annotation` or `automation paused: <pause-reason>` |
 | `resource-quota-exceeded` | pre-check | `spec.safetyChecks.enableResourceQuotaFilter` | Blocks if projected pod resources would exceed applicable `ResourceQuota` | no retry from check itself | `resource quota exceeded (ResourceQuota/<name>)` |
 | `requests-exceed-limits` | final plan consistency pre-check | always on | Blocks when the filtered resize plan would leave a desired request above the effective desired/current limit | no retry from check itself | `desired request <x> exceeds limit <y> (container=<name>, resource=<resource>)` |
 | `min-ready-duration-not-met` | final health pre-check | `spec.safetyChecks.minReadyDuration` | Blocks until pod has been Ready for at least `minReadyDuration` | retryable: uses remaining ready time when pod is Ready but elapsed Ready time is below `spec.safetyChecks.minReadyDuration`; uses `spec.safetyChecks.resizeRetryInterval` when pod is not Ready or ready transition time is unknown | `pod not ready` or `pod ready for ... (<...)` |
@@ -55,6 +55,7 @@ How to interpret `retry` in the context of this document:
 | `ceiling-clamped` | plan build | `spec.enablement.*.(requests|limits).ceiling` | Clamps a desired value down to the configured ceiling before execution | filter metadata includes `value`, `originalDesired`, `clampedDesired`, and `rule` | n/a (summary marker) |
 | `automation-strategy-disabled` | action filter | `spec.enablement.*` fields in the referenced strategy | Removes actions disallowed by direction such as `upsize`, `downsize`, or `setFromUnspecified` | filter metadata may include `direction` | `upsize disabled`, `downsize disabled`, or `setFromUnspecified disabled` |
 | `change-below-threshold` | pod action filter | `spec.safetyChecks.minCpuChangePercent`, `spec.safetyChecks.minMemoryChangePercent` | Removes actions whose percent delta is below threshold | target list only | `delta ... below minimum ...` |
+| `container-skip-active` | pod action filter | always on | Removes actions for containers listed in `rightsizing.kubex.ai/skip-containers`. Pod annotation key wins over owners; empty pod value means skip none for that pod. If pod key is absent, nearest supported owner with non-empty value wins. Empty owner values are ignored. No merges across pod/owners or multiple owners. Namespace scope is not supported. | filter metadata includes `scope=container` and `container=<name>` | `container skipped by annotation` |
 | `hpa-resource-managed` | pod action filter | `spec.safetyChecks.enableHpaFilter` | Removes actions for CPU or memory managed by a matching HPA, including KEDA-managed HPA handling | filter metadata may include `source=hpa` and `hpaMode` | `HPA targets <resource>` |
 | `vpa-resource-managed` | pod action filter | `spec.safetyChecks.enableVpaFilter` | Removes actions for resources with an active recommendation from a matching VPA | target list only | `VPA manages <resource>` |
 | `node-capacity-insufficient` | pod action filter | `spec.safetyChecks.requireNodeAllocatable`, `spec.safetyChecks.nodeCpuHeadroom`, `spec.safetyChecks.nodeMemoryHeadroom` | Removes request increase actions when remaining allocatable capacity, after headroom reservation, is insufficient | target list only | `remaining <x> < delta <y>` |
@@ -67,7 +68,7 @@ How to interpret `retry` in the context of this document:
 1. Build candidate actions from selected recommendations.
 2. Apply `automation-strategy-disabled`.
 3. Run early pre-checks: `pod-terminating`, `namespace-protected`, optional `pause-active`, optional `resource-quota-exceeded`.
-4. Apply pod action filters: `change-below-threshold`, `hpa-resource-managed`, `vpa-resource-managed`, `node-capacity-insufficient`, `pod-limit-range-violated`.
+4. Apply pod action filters: `change-below-threshold`, `container-skip-active`, `hpa-resource-managed`, `vpa-resource-managed`, `node-capacity-insufficient`, `pod-limit-range-violated`.
 5. Run final non-retryable plan consistency checks: `requests-exceed-limits`.
 6. Run final retryable health checks: `min-ready-duration-not-met`, `owner-pods-not-ready`, `max-unavailable-exceeded`.
 7. Execute in-place resize or eviction if actions remain.
@@ -84,11 +85,13 @@ Notes:
 - `appliedFilters` contains pruned actions with `name`, optional filter `metadata`, and `targets` with `container`, `usage`, and `resource`.
 - Bounds enforcement also appears in `appliedFilters`: `floor-clamped` and `ceiling-clamped` indicate the controller adjusted the desired value before deciding whether any resize action still remained.
 - `pause-active` sets `failedChecks[].metadata.scope` to `pod` or `namespace`; namespace pauses also include `failedChecks[].metadata.namespace`.
+- `container-skip-active` sets `appliedFilters[].metadata.scope=container` and `appliedFilters[].metadata.container=<name>`.
 
 Example interpretation:
 
 - `{"name":"min-ready-duration-not-met","message":"pod not ready"}` in `failedChecks` means execution is blocked for now and retried.
 - `{"name":"resource-quota-exceeded","message":"resource quota exceeded (ResourceQuota/team-quota)","metadata":{"name":"team-quota"}}` in `failedChecks` identifies the specific blocking quota.
+- `{"name":"container-skip-active","metadata":{"scope":"container","container":"app"},"targets":[{"container":"app","usage":"requests","resource":"cpu"}]}` in `appliedFilters` means only that container action was removed because `skip-containers` matched that container.
 - `{"name":"hpa-resource-managed","targets":[{"container":"app","usage":"requests","resource":"cpu"}]}` in `appliedFilters` means that resize action was removed because HPA controls that resource.
 - `{"name":"floor-clamped","metadata":{"value":"200Mi","originalDesired":"128Mi","clampedDesired":"200Mi","rule":"AutomationStrategy/example-rule"},"targets":[{"container":"app","usage":"requests","resource":"memory"}]}` in `appliedFilters` means the recommendation was raised to the strategy floor before execution.
 

@@ -105,7 +105,8 @@ kubectl get events -A --field-selector reason=PolicyEvaluationEvictResize
 
 - no valid Kubex recommendation for the target container
 - recommendation older than `maxAnalysisAgeDays`
-- pause annotation present
+- pod-level pause annotation present, which blocks whole pod
+- container-level pause annotation present, which can partially filter multi-container pods while leaving sibling actions eligible
 - HPA or VPA overlap
 - ResourceQuota or LimitRange conflict
 - node allocatable headroom gate