Optimizer Pod Stuck in ImagePullBackOff Due to Docker Rate Limits

[shreyabiradar07]

Description

The kruize-optimizer pod intermittently fails to start and gets stuck in ImagePullBackOff state due to Docker Hub's unauthenticated pull rate limits being exceeded.

Environment

• Namespace: monitoring
• Image: busybox:1.36
• Cluster: Minikube

Current Behavior

The pod's init container fails to pull the busybox:1.36 image from Docker Hub used in the init container, resulting in the following error:

429 Too Many Requests
toomanyrequests: You have reached your unauthenticated pull rate limit.

InitContainers: []corev1.Container{
						{
							Name:  "wait-for-kruize",
							Image: "busybox:1.36",
							Command: []string{
								"sh",
								"-c",
								`echo "Waiting for Kruize service to be ready...";
								until wget -q -O- --timeout=5 http://kruize:8080/health >/dev/null 2>&1 || wget -q -O- --timeout=5 http://kruize:8080 >/dev/null 2>&1; do
								echo "Kruize service not ready yet, waiting...";
								sleep 5;
								done;
								echo "Kruize service is ready!";`,
							},
						}

Pod Status

NAME                               READY   STATUS                  RESTARTS   AGE
kruize-5b65cff76b-sv5kt                1/1     Running                 0          48m
kruize-db-deployment-c8f9877b-vg7hk    1/1     Running                 0          48m
kruize-operator-67f7d84d78-pmz7q       1/1     Running                 0          48m
kruize-optimizer-79cf8dcfbf-bpwgv      0/1     Init:ImagePullBackOff   0          48m
kruize-ui-nginx-66ddc96897-wkrfx       1/1     Running                 0          48m

Error Events

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  48m   default-scheduler  Successfully assigned monitoring/kruize-optimizer-79cf8dcfbf-bpwgv to minikube
  Warning  Failed     47m   kubelet            Failed to pull image "busybox:1.36": failed to pull and unpack image "docker.io/library/busybox:1.36": failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://registry-1.docker.io/v2/library/busybox/manifests/sha256:b7f3d86d6e84fc17718c48bcde1450807faa2d56704205c697b4bd5df7b9e29f: 429 Too Many Requests
toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
  Normal   Pulling  44m (x5 over 48m)  kubelet  Pulling image "busybox:1.36"
  Warning  Failed   44m (x4 over 48m)  kubelet  Failed to pull image "busybox:1.36": failed to pull and unpack image "docker.io/library/busybox:1.36": failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://registry-1.docker.io/v2/library/busybox/manifests/sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662: 429 Too Many Requests
toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
  Warning  Failed   44m (x5 over 48m)      kubelet  Error: ErrImagePull
  Normal   BackOff  3m26s (x188 over 48m)  kubelet  Back-off pulling image "busybox:1.36"
  Warning  Failed   3m15s (x189 over 48m)  kubelet  Error: ImagePullBackOff

Suggested Fix

Use quay based image replacing the current docker busybox image or alternatively can use Kruize quay image in the init container