feat: validate worker operational readiness

[bugkeep]

Fork CI experiment

This draft validates the revised worker operational-readiness implementation before any upstream PR is reopened or updated.

Dependency

• base: fix/ci-flannel-e2e-pin (fork CI fix corresponding to merged upstream feat: validate Helm release readiness #110)
• head: feat/worker-operational-validation (candidate update for upstream fix: preserve Kine revision ordering under concurrent writes #105)
• do not merge this experiment

What is validated

• quarantine workers at kubelet registration and remove the taint only after readiness succeeds
• wait for read-only prerequisites once, then run storage, scheduler, and service probes as bounded one-shot phases
• use the cluster default StorageClass and preserve user-managed local-path deployments
• wait for EndpointSlice publication and stale EndpointSlice cleanup before service validation
• validate cluster DNS with the canonical FQDN kubernetes.default.svc.cluster.local.
• validate direct PodIP and Service ClusterIP routing
• keep cleanup bounded after parent cancellation and surface missing machine-state persistence
• tolerate concurrent CoreDNS Service deletion

Root-cause evidence

Diagnostic run 30032506707 showed:

• Pod /etc/resolv.conf: correct search domains and nameserver 10.43.0.10
• canonical DNS FQDN via configured resolver: PASS
• canonical DNS FQDN via kube-dns Service IP: PASS
• canonical DNS FQDN via direct CoreDNS EndpointSlice address: PASS
• direct application PodIP: PASS
• application Service ClusterIP: PASS
• partially qualified kubernetes.default.svc: NXDOMAIN under BusyBox nslookup

The failure was a probe false negative caused by querying a partially qualified name, not a CoreDNS, kube-proxy, Flannel, or Service data-plane failure.

OCR notes

• fixed: prerequisite wait has its own deadline
• fixed: stale EndpointSlices cannot satisfy a new probe run
• fixed: control-plane nodes cannot receive the worker bootstrap taint
• fixed: service-probe attempts are bounded and terminal errors include Pod state/logs
• retained intentionally: same-node service fallback is required for the first worker
• retained intentionally: cleanup uses an independent bounded context so cancellation does not leak probe resources

Validation

• local: go test ./deploy ./server ./object -count=1
• fork CI: all jobs passed in run 30033343737
  • Running Go tests
  • Go-Linter
  • Back-end
  • Front-end
  • UI tests, including the worker VM readiness path
• no new *_test.go files are included in the published diff

Fix: casosorg#101