Skip to content

fix(runtime): share the failure classifier across containerd and the microVM runtimes - #198

Merged
Shine-neko merged 1 commit into
mainfrom
fix/shared-failure-classifier
Jul 25, 2026
Merged

fix(runtime): share the failure classifier across containerd and the microVM runtimes#198
Shine-neko merged 1 commit into
mainfrom
fix/shared-failure-classifier

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Only the Docker runtime used the shared retry-vs-terminal classifier. Cloud Hypervisor and Firecracker each carried a local, divergent copy of classify_vm_start_error, and containerd had no classification at all. All of them now go through src/hypervisor/classifier.rs.

Unifying them surfaced four pre-existing defects, fixed here:

  • The Firecracker worker set a failure status without ever moving restart_count. Since the reconcile loop keeps polling that status, a permanently failing boot was retried on every tick with no bound.
  • Firecracker's local copy had dropped the missing-firmware case, so it consumed the whole restart budget instead of failing fast.
  • Both microVM runtimes treated a missing config, and a rejected instance spec, as retryable — unlike Docker.
  • containerd had no fail-fast path, and mapped a missing config to the generic Error status instead of config_error.

The classify-then-apply sequence is factored into apply_vm_start_failure(), used at the four microVM call sites. Its invariant is that restart_count moves on either branch, which is what closes the unbounded retry.

One deliberate exception: containerd keeps InstanceCreationFailed retryable. Docker only raises that variant when the daemon rejects a container spec, but containerd wraps every gRPC call in the create path with it, so it also covers a momentarily unavailable shim. Failing fast there would abandon recoverable deployments.

Docs: documented the fail-fast behaviour in the status lifecycle page, and fixed a contradiction between two pages on how labels: are handled on the microVM runtimes (they are stored and filterable as Ring metadata, just not applied to the VM).

Tests

764 unit tests pass, including 10 new ones covering the shared classifier, the counter invariant, and the containerd exception. cargo fmt and cargo clippy --all-targets are clean.

The e2e suites were not run: they need a host with KVM and the runtime binaries available.

@Shine-neko
Shine-neko merged commit ddbf016 into main Jul 25, 2026
5 checks passed
@Shine-neko
Shine-neko deleted the fix/shared-failure-classifier branch July 25, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant