Error from server: Get "https://192.168.90.20:10250/containerLogs/run-hlab/hlab-pg4j9-runner-zw7rs/runner?follow=true": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-02-02T09:42:59Z is after 2026-02-01T21:27:44Z
Cluster status:
NAMESPACE NAME READY STATUS RESTARTS AGE
arc arc-gha-rs-controller-f9c6b7699-2nt7j 1/1 Running 0 250d
... <other pods omitted for brevity> ...
run-hlab hlab-pg4j9-runner-zw7rs 2/2 Running 0 14h
Certificate expiry errors also appeared in k3s system logs:
sudo systemctl status k3s
Feb 02 09:56:43 run k3s[1248392]: E0202 09:56:43.450210 1248392 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2026-02-02T09:56:43Z is after 2026-02-01T21:27:44Z"
Workaround
On run.l:
sudo k3s certificate rotate
sudo systemctl restart k3s
Output confirming rotation:
INFO[0000] Server detected, rotating server certificates
...
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1770026072, please restart k3s server or agent to rotate certificates
On env-ci-1:
sudo systemctl restart k3s-agent
After that:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
env-ci-1.l Ready <none> 388d v1.28.5+k3s1
run Ready control-plane,etcd,master 2y17d v1.28.5+k3s1
...
kubectl get pods -A
run-hlab hlab-pg4j9-runner-d49mp 2/2 Running 0 87s
Cluster status:
Certificate expiry errors also appeared in k3s system logs:
Workaround
On run.l:
Output confirming rotation:
On env-ci-1:
After that: