Possibility to pass vm configuration?

[manuschillerdev]

Is it somehow possible to pass detailed VM configuration to the VM? I understand that apple containers uses kata - can we fully configure it?

My concrete usecase would be to alter the network configuration of a vm so all traffic gets routed through a proxy on the host.

[jglogan]

You want all traffic to go through a network proxy, right? Not just HTTP/HTTPS traffic?

There's a couple different layers involved in the answer to this question.

vminitd in the containerization library is where you have runtime control over what's happening in the VM:

• The vminitd API determines what can be controlled from the host.
• vminitd doesn't do anything with iptables/nftables today, and one challenge in doing so is that vminitd is pretty much pure Swift, so you'd either need the appropriate tool in the image for it to work (not guaranteed) or implement the nettlink support required to do the rule programming.
• The SandboxService in container-runtime-linux is what interacts with the vminitd API to manage its container.
• The container-apiserver interacts with SandboxService via its XPC API.
• The container tool uses ContainerClient to interact with the API server via its XPC API.

How much work that would need to be done to support this use case would depend on how far up the stack it needs to be plumbed.

[manuschillerdev]

Thanks! I currently fiddle with a sandbox solution on top of dev containers. It would be totally fine if I could for example load ebpf modules on startup for any kind of proxy work as well. But afaik it is not possible to interact with the vm at all besides loading a totally custom kernel, right? Like changing the config or having options similar to "colima ssh"

[jglogan]

But afaik it is not possible to interact with the vm at all besides loading a totally custom kernel, right?

You can interact with the guest VM but those actions are specified entirely by what is in the vminitd API.

You can definitely specify a custom kernel, but that's not interacting with a VM in the dynamic sense.

[manuschillerdev]

I've been tinkering with ways to extend container with a primitive for interacting with the VM layer. One approach: expose the internal init filesystem as a user-configurable option via --init-image.

Sketch/PoC: https://github.com/manuschillerdev/container/pull/1/files

What this enables:

• Custom boot-time logic before the OCI container starts
• Running additional daemons (e.g., daemons to configure eBPF network filters, logging agents) inside the VM
• Debugging/instrumenting the init process

In my initial attempts, the custom image wraps the default vminitd, so it's currently behaving like an entrypoint.
Curious if this direction aligns with the project's goals, or if there's a better abstraction for these use cases @jglogan ?
If there is interest, I would add this as proper PR with tests and doc against this repository.

Example for a minimal entry point binary

// custom-init/wrapper.go
package main

import (
    "fmt"
    "os"
    "syscall"
)

func main() {
    kmsg, err := os.OpenFile("/dev/kmsg", os.O_WRONLY, 0)
    if err == nil {
        kmsg.WriteString("<6>custom-init: === CUSTOM INIT IMAGE RUNNING ===\n")
        kmsg.Close()
    }

    err = syscall.Exec("/sbin/vminitd.real", os.Args, os.Environ())
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to exec vminitd.real: %v\n", err)
        os.Exit(1)
    }
}

Containerfile:

# custom-init/Containerfile
FROM ghcr.io/apple/containerization/vminit:0.12.1 AS base

FROM ghcr.io/apple/containerization/vminit:0.12.1
COPY --from=base /sbin/vminitd /sbin/vminitd.real
COPY wrapper /sbin/vminitd

• build the binary with cd custom-init && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o wrapper wrapper.go
• build the image with bin/container build -t local/custom-init:0.1.2 custom-init/
• run the container with our custom init-image: bin/container run --name init-test --init-image local/custom-init:0.1.2 alpine:latest echo "hello"
• check the logs with cat "/Users/$USER/Library/Application Support/com.apple.container/containers/init-test/vminitd.log" | head -50

[manuschillerdev]

@jglogan @dcantah Sorry for pinging you directly again. I am not sure if the option to customize the boot process would be interesting for the project and if the solution I landed on (#937) would be a good fit for the overall architecture. Quick feedback on whether pursuing this idea further was worth it would be appreciated. Of course, please also let me know any time if I just missed sth in the contribution guidelines - I'll gladly resolve any issues with the PR. Thanks!

[camparker]

@manuschillerdev

(Apologies for such a basic question) I'm interested in using your feature for creating custom init images, and I was wondering if you could provide an example that incorporates an added service or daemon to the custom init image?

This is not my usual area of focus, so I'm unsure how to really go about doing this. So far I have followed the example shown in the How To documentation, but I'm seeing that the base init image lacks the system packages needed to install anything.

I appreciate any help you can provide, and thanks for the feature addition!

[jglogan]

The enhancement from @manuschillerdev was merged as #937. The issue provides the overview of what you can do.

You're not just going to be able to install whatever you want. This is a microVM approach, where all that is running under the workload is a direct-boot linux kernel and a single guest agent process. What @manuschillerdev demonstrates is how to build a wrapper around the guest agent to add capabilities.

[camparker]

Thank you for the reply @jglogan

I dug a little deeper after my previous message and I think I have a better understanding of the feature/use case now. I should have edited my message to indicate so.

[jglogan]

Glad you're making progress! It'd be cool if you find you're able to use the feature, @manuschillerdev put a decent amount of work into it!