fix(runtime): share the failure classifier across containerd and the microVM runtimes - #198
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only the Docker runtime used the shared retry-vs-terminal classifier. Cloud Hypervisor and Firecracker each carried a local, divergent copy of
classify_vm_start_error, and containerd had no classification at all. All of them now go throughsrc/hypervisor/classifier.rs.Unifying them surfaced four pre-existing defects, fixed here:
restart_count. Since the reconcile loop keeps polling that status, a permanently failing boot was retried on every tick with no bound.Errorstatus instead ofconfig_error.The classify-then-apply sequence is factored into
apply_vm_start_failure(), used at the four microVM call sites. Its invariant is thatrestart_countmoves on either branch, which is what closes the unbounded retry.One deliberate exception: containerd keeps
InstanceCreationFailedretryable. Docker only raises that variant when the daemon rejects a container spec, but containerd wraps every gRPC call in the create path with it, so it also covers a momentarily unavailable shim. Failing fast there would abandon recoverable deployments.Docs: documented the fail-fast behaviour in the status lifecycle page, and fixed a contradiction between two pages on how
labels:are handled on the microVM runtimes (they are stored and filterable as Ring metadata, just not applied to the VM).Tests
764 unit tests pass, including 10 new ones covering the shared classifier, the counter invariant, and the containerd exception.
cargo fmtandcargo clippy --all-targetsare clean.The e2e suites were not run: they need a host with KVM and the runtime binaries available.