Feature Discussion: distro subsystem

[dcantah]

Background

Due to the popularity of projects like WSL, Lima and many others in this realm, it's clear that there's appetite for a container/VM hybrid solution that plops you in a Linux environment as fast as possible, while being effortless to get started, highly customizable, and integrated with your host system in clever ways.

While our project is currently tailored around running single containers, and that will continue to be the main focus, we've been mulling over a new experimental surface that would complement this functionality and that we already have most of the building blocks for from the rest of the project. We've been thinking of naming this new feature distro.

What is distro?

The distro feature would provide a more full featured VM-like experience, giving you (subject to change, but our current thoughts):

• Instant Linux environments: Boot into Ubuntu, Debian, Alpine, or any other distribution in seconds. The distributions would just be container images like we're all used to, but the main difference being the set to choose from would be more full featured images with non-stripped down userlands, and we'd by default try and run any init system in the image (/sbin/init).

• Persistent workspaces: Unlike ephemeral containers, distros maintain state between sessions. Writes are persistent in other words.

• Seamless integration:

  1. DNS name by default on every distro, making talking to network services inside a breeze.
  2. Host filesystem access by default. /Users exposed in the distro as /Users (or wherever you please). Read-only by default, but we would have ample flags to tweak where to expose this, r/o vs. r/w, and many others.
  3. Automatic ssh agent forwarding.
  4. Automatic user creation. Every distro launched would have a default user created that is named after your macOS username.
  5. (TBD) Run macOS binaries in the distro. Through some clever tricks with binfmt and a process on the host that we can talk to, we can launch macOS binaries on the host and relay the output to your distros shell so you don't need to leave the terminal pane you're working in to run a quick binary on the host.

• Simple management: Straightforward commands to manage your Linux environments.

Interface brainstorm

We're envisioning commands like the following:

container distro list                                                # List available distributions
container distro create mycoolregistry.foo/ubuntu:24.04 ubuntu24.04  # Create a new distro from an image.
container distro run --name ubuntu24.04                              # Start a distro (with no arguments gives a login shell)
container distro run --name ubuntu24.04 bash                         # Run a single command in a distro
container distro shutdown ubuntu24.04                                # Gracefully shutdown the distro and virtual machine
container distro remove ubuntu24.04                                  # Remove a distro
container distro set-default alpine                                  # Set the default distro used for `run` if `--name` isn't provided

Questions for the Community

We'd love your feedback:

1. Is this something you'd use? Would this be valuable to your workflow?
2. What features are must-haves?

• Automatic port forwarding?
• Shared filesystem mounts?
• Snapshot/backup capabilities?

3. How would you use this differently from containers? What use cases would this unlock for you?

---

Please share your thoughts! Whether you'd be an eager early adopter, or you think this doesn't fit the project's direction 😄, we want to hear from you.

[afbjorklund]

This feature is useful in WSL (distro integration, port forwarding, filesystem sharing), and something also Lima does.

In fact you can run Lima with the WSL2 driver, and it will do the user creation and those other missing pieces for you...
You can do it yourself, but then you also have to do the provisioning and forwarding those other things in the template.

Nowadays WSL even does GUI (with WSLg), and that would be the real killer feature for this "distro" integration as well?

Meanwhile we have an experimental AC/DC driver so that you can use Lima to customize your "container" (Apple or Kata)
When you do the container run or docker run you get root by default, so we create the user and the mounts and so on.

• The Lima container drivers lima-vm/lima#3829

[afbjorklund]

Since virtiofs mounts were broken in container at the time I was trying it out, I went with the default reverse-sshfs mounts (that only require fuse on client and sftp on host). But now it should be able to just do -v /Users:/mnt/Users instead.

Also most external drivers are not fully working yet after the major upgrade from Lima 1.2 to Lima 2.0 that just happened, so only the WSL2 driver is fully operational when it comes to containers. The QEMU and VZ drivers do work for virtual machines.

[saehejkang]

If distro is meant to be a "container/VM hybrid", would having some sort of "consoling" connection be in scope of this work? If the user spins up the a container with something like a CentOS image, should there not be a way to connect to the console in the web browser? Maybe using websockets and noVNC? This is all dependent on if the container images actually have GUI support and I know that it was mentioned above for WSL.

Nowadays WSL even does GUI (with WSLg), and that would be the real killer feature for this "distro" integration as well?

But what about GUI support for lima? I found that there is a project called lima-gui. Funny enough, I realized that @afbjorklund is the creator of it so maybe he can speak to it more on how feasible integration is?

[afbjorklund]

It is possible to display a SSH terminal connection in a web browser, using something like xTerm.js and ssh...
There is a prototype (non-production) integration here: https://github.com/afbjorklund/go-ssh-web-client (WebSocket)

You can also show a VNC graphical connection in the web browser, using something like noVNC and vnc...
At one point I was thinking of bundling these tools with Lima, but in the end we did not. lima-vm/lima#1003 (comment)

So currently we just use the external ssh and vncviewer.

The user can still start the web clients, instead of those.

But what about GUI support for lima?

There is built-in support, for opening a display in the Linux VM. You can either use the default from the hypervisor, which looks like a regular window. But on some platforms, that might consume a lot of resources (so the default is to keep it hidden).
Then there is support for starting a VNC server instead. The Lima GUI application has a built-in SSH client and a built-in VNC viewer, so it connects using these. But it will be in a window, and the graphic will be started from the client (not the server).

• Providing graphical lima example lima-vm/lima#1002

"Hello World": limactl start template:experimental/vnc

With the built-in support in "WSLg", you can start an X11 application from the Linux command line and the application will show up next to the native Windows host applications. There is no RDC or anything needed, it is integrated with the desktop GUI.

https://github.com/microsoft/wslg

It's not a normal "container" feature.

[afbjorklund]

I don't see why you need to use new commands, for these "distro" containers?

container distro run --name ubuntu24.04 # Start a distro (with no arguments gives a login shell)
container distro run --name ubuntu24.04 bash # Run a single command in a distro

container distro shutdown ubuntu24.04 # Gracefully shutdown the distro and virtual machine
container distro remove ubuntu24.04 # Remove a distro

Shouldn't these just use "stop" and "rm" commands, like with any other container.

Commands such as "list" and "create" are still convenient wrappers, to the usual.

[jglogan]

It's a good point. It's really create that contains the specialization for distro.

And distro run is just a wrapper that invokes the distro create instead of the normal one.

Seems the important consideration is whether start needs to do anything special. Ideally everything unique to distro can be housed in container creation, and in a specialized runtime helper (container-runtime-linux-distro) if necessary.

[afbjorklund]

It is possible to add a lot of new command wrappers, the way it has been done in distrobox and Podman toolbx.
But the important ones are "create" (run) and "enter" (exec), that is where the magic flags are hidden/needed:

container=$(container run -d ...)
container exec -it $container bash

[afbjorklund]

Every distro launched would have a default user created that is named after your macOS username.

A simple approach to get started with the idea, would be to prompt the user (after starting the container):

https://learn.microsoft.com/en-us/windows/wsl/setup/environment#set-up-your-linux-username-and-password

But a typical container would have a custom image for the user, with the account and the home mountpoint dir:

RUN groupadd -g $gid $group && useradd -u $uid -g $gid -d $home -s $shell -m -N $user \
&& rm -f /etc/.pwd.lock
VOLUME $home
WORKDIR $home
USER $user
CMD ["$shell"]

To not have to do it later at runtime, similar to how packages are installed in the image and not in the container?

Lima uses cloud-init and cloud-config.yaml for the regular cloud images, but it is overkill for container images...

[StephenRudolph]

"Is this something you'd use? Would this be valuable to your workflow?"

Yes!

"What features are must-haves?"

• The option to use SSH agent forwarding, so that the Linux VM has access to my host's Git credentials, making it easy to even push signed commits
• Coming from a corporate environment perspective, the option to automatically include certificates from the host in the VM's trust store
• "Automatic port forwarding?" - Maybe - I'd typically run VS Code's Remote SSH extension, which would already provide port forwarding, wouldn't mind the option though!
• "Shared filesystem mounts?" - Yes!
• "Snapshot/backup capabilities?" - No - even for a persistent workspace, I would still have it defined in code so that rebuilding it is not an issue

"How would you use this differently from containers? What use cases would this unlock for you?"

Some examples:

• Having a full environment for each project while keeping my host machine "clean" - this would be especially nice when working with databases where I'd like some amount of persistence
• Running MCP servers for LLMs - I'd be much more comfortable giving an MCP server access to a VM than my host machine
• Running as a standard user account on my host while having the ability to access a full development environment where I have root access
  • This would benefit many corporate scenarios and I suspect education as well

[dcantah]

Thanks for the feedback :)

[subpop]

I independently started experimenting with this very idea, building on top of container as the underlying runtime, when I happened upon this discussion thread. I love this idea, and would love to see it natively supported in container as well!

I continued my experimentation anyway to see what I could come up with. My goal was to be 100% compatible with the existing toolbox out-of-the-box. This felt like an important feature to me, so that existing efforts from other communities can overlap with this one.

I found that most of the customization happens during the container creation, and passing in some values from the host at "enter" was important as well (specifically the shell, as many Linux distributions don't ship zsh). Toolbox also does a lot of bind mounting of host devices like the D-Bus socket and the SSH agent socket that is a lot more difficult to achieve in this case because we're actually crossing two boundaries (macOS -> guest virtual machine -> container). I ended up not setting the user's host home directory to the container's home directory, but I still mounted it into the container. I found that I had some macOS binaries that execute when shell logs in that don't work in a Linux container (obviously!). This was a really fun experiment though. I hope this feature gains some traction!

https://github.com/subpop/applebox

[afbjorklund]

That is the approach used by Lima, it uses a persistent guest home - and mounts the host home as the working directory.

The default mount will mount the host home directory as read-only in the guest, as well: https://lima-vm.io/docs/examples/

[subpop]

Oh neat, I never thought to check if Lima used persistent guest homes. I know it uses a separate /home/$USER.linux, but that makes sense to persist it. I've been a Lima user for a number of years.

[dcantah]

I'd need to read more on what the tricks with binfmt might be in order to invoke macOS binaries on Linux and get them to pass through to the host

This is very hairbrainy, but theoretically it should be possible (given we'd share in the host files over virtiofs) to register a binfmt handler that has the macho headers written to it and assign some program we write that exists in the guest as the emulator, have some process/service listening on the host over a vsock port that the emulator program in the guest dials out to, and then forward what the user tried to run in the guest out to this service over vsock (arguments, envvars, etc.). The service on the host would actually run the program and just stream output back into the guest to make it appear like it was ran in the guest. The tricky part there is how paths would be treated.

[subpop]

ChatGPT was probably a bit elaborate in its rendering of the scheme, but this is what comes to mind... 🤣

[dcantah]

LOL. Seems about right