Skip to content

Conversation

@goyalpalak18
Copy link

Description

I fixed a resource leak where network interfaces (TAP devices) and Traffic Control (TC) rules were being left behind if a VMM crashed or was killed externally.

The issue was a logic flaw in how we handled the Kill() command. Previously, the code tried to stop the VMM process first. If that failed—for example, if the process was already dead (returning ESRCH)—the function would return an error immediately and skip the network.Cleanup() step entirely.

This was causing serious issues in crash-loop scenarios (like an OOMing unikernel). Since the cleanup never ran, tap0_urunc devices and TC redirect filters kept piling up in the network namespace, eventually causing network blackholes and exhausting node resources.

Changes

  • pkg/unikontainers/unikontainers.go:

    • I decoupled the VMM stop operation from the network cleanup.
    • The code now guarantees that network.Cleanup("tap0_urunc") runs, even if vmm.Stop() returns an error. If the stop fails, I log a warning but still proceed to clean up the network resources before returning.
  • pkg/unikontainers/hypervisors/utils.go:

    • I updated the killProcess function to handle syscall.ESRCH gracefully.
    • If we try to kill a process that is already dead, I now treat that as a success (returning nil) instead of an error. This makes the stop operation idempotent—if it's dead, my job is done.

Testing

I verified this using a "zombie VMM" reproduction scenario:

  1. Reproduction:

    • I started a container, then manually killed the VMM process using kill -9.
    • I ran urunc kill <container_id>.
    • Before: The command failed with "no such process," and ip link show proved the TAP device was still there.
    • After: The command succeeded (silently handling the dead process), and I confirmed the TAP device was correctly removed.
  2. Crash Loop:

    • I simulated a Kubernetes crash loop and verified that after 10+ restarts, tc filter show showed zero orphaned rules.

Impact

  • Stability: This prevents network degradation in long-running clusters where pods might crash and restart frequently.
  • Resource Management: We now ensure strict cleanup of kernel network objects regardless of whether the application crashed or stopped gracefully.
  • Correctness: The kill command now accurately reflects that the resources are gone, rather than erroring out just because the process was already missing.

Signed-off-by: goyalpalak18 <goyalpalak1806@gmail.com>
@netlify
Copy link

netlify bot commented Jan 25, 2026

Deploy Preview for urunc ready!

Name Link
🔨 Latest commit ce7e5c0
🔍 Latest deploy log https://app.netlify.com/projects/urunc/deploys/6977ab42f0c0d2000809c844
😎 Deploy Preview https://deploy-preview-402--urunc.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@cmainas
Copy link
Contributor

cmainas commented Jan 26, 2026

Hello @goyalpalak18 ,

thank you for this contribution. Please create an issue before opening a PR. May I ask which network namespace you set during your test?

@goyalpalak18 goyalpalak18 force-pushed the fix/network-cleanup-on-vmm-crash branch from 20cd951 to ce7e5c0 Compare January 26, 2026 17:58
@goyalpalak18
Copy link
Author

@cmainas Thanks! I have created the tracking issue as requested: #408

To answer your question: I tested this using the standard CNI setup, so the TAP device was located inside the pod's network namespace.

(Note: I also force-pushed the branch just now to remove some unrelated commits that accidentally got mixed in. The PR is clean now.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants