Skip to content

feat: add dry-run mode - #197

Open
vadossam wants to merge 3 commits into
ctrox:mainfrom
vadossam:add-dry-run
Open

feat: add dry-run mode#197
vadossam wants to merge 3 commits into
ctrox:mainfrom
vadossam:add-dry-run

Conversation

@vadossam

Copy link
Copy Markdown

Motivation

Enabling zeropod on an existing workload is currently an all-or-nothing decision. To find out how a pod would actually behave - how often it would scale down, whether scaledown-duration is tuned right, how restore latency looks on first connection - you have to let it really checkpoint and restore, which is risky to try on production or production-like workloads.

This PR adds a dry-run mode so you can observe zeropod's scale-down/restore decisions on a real workload without it ever actually happening.

What this adds

zeropod.ctrox.dev/dry-run: "true"

With it set, the shim runs its scale-down timer exactly as normal - same activity tracking, same scaledown-duration logic - but when the timer fires it does not checkpoint/kill the process and does not enable the eBPF traffic redirect. The real process keeps running and serving traffic the entire time. Instead:

  • it logs would have scaled down / would have restored,
  • emits a matching Kubernetes Event on the pod,
  • increments zeropod_dry_run_scale_downs_total / zeropod_dry_run_would_restores_total (same label set as the existing checkpoint/restore metrics).

Restore detection covers both of the triggers real zeropod supports:

  • TCP activity, via the existing activity tracker (lastActivity()/socket_tracker) - polled once a second while in the simulated scaled-down state.
  • kubectl exec, mirrored from the real restore-on-exec path in wrapper.Exec, including the same suppression semantics: an active exec session keeps blocking the simulated scale-down for its whole duration, not just at the moment exec was called (rescheduling is deferred to wrapper.Delete's existing runningExecs == 0 check, exactly like the real path).

Cannot be combined with migrate/live-migrate (rejected at config-parse time) - the two features are contradictory by definition, and node-drain eviction (shim/evac.go) checkpoints independently of the normal timer path this feature guards.

Design notes

The core constraint driving the implementation: dry-run must never mutate real container state. Container.scaledDown stays false for the container's entire life while dry-run is active, tracked instead via a separate dryRunScaledDown flag. This matters because several other things key off ScaledDown():

  • manager/pod_scaler.go would shrink cpu/memory requests on a container that's actually still fully running.
  • manager/pod_labeller.go would flip the pod's status.zeropod.ctrox.dev/<container> label to SCALED_DOWN, misleading anything reading it as ground truth.
  • Pids/Stats/Kill in the task service branch on it to synthesize scaled-down responses.

So instead of adding a new ContainerPhase, the new ContainerStatus.dry_run field is a side-channel: event_creator.go checks it before the phase-based switch and creates a distinct Dry run event, while phase itself never changes - pod_scaler/pod_labeller needed no changes at all.

Skipping the real scale-down is a single early branch in scaleDown(), before activator.Reset() (the eBPF redirect enable) is ever called - confirmed via the eBPF source that track_activity() runs on ingress independently of and before the redirect-enable check, so the activity tracker keeps working correctly with the redirect never enabled.

Files touched

  • api/shim/v1/{config.go,shim.proto} - annotation + two new fields (ContainerStatus.dry_run, ContainerMetrics.dry_run_scale_downs/dry_run_would_restores), regenerated via make ttrpc
  • shim/dryrun.go (new) - the whole mechanism
  • shim/checkpoint.go, shim/container.go, shim/task/service_zeropod.go - wiring
  • manager/event_creator.go, manager/metrics_collector.go - Events and metrics
  • docs/configuration/README.md, docs/metrics.md

Testing

Unit tests for config parsing (including the migrate-conflict rejection) and event creation, plus an e2e case asserting the pod's status label never leaves RUNNING under dry-run.

Manually verified end-to-end on a real RKE2 cluster (in addition to local kind testing) - actual kubectl describe pod output:

Normal  Created    2m24s  kubelet                    Created container: nginx
Normal  Started    2m24s  kubelet                    Started container nginx
Normal  Dry run    2m8s   zeropod.ctrox.dev/manager  Dry-run: container nginx would have scaled down after 15s of inactivity
Normal  Dry run    23s    zeropod.ctrox.dev/manager  Dry-run: container nginx would have restored (last activity 836.526562ms ago)

The container never actually restarted throughout (0 restarts, status label pinned at RUNNING) while these events were produced from real, unmodified traffic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant