Skip to content

svm_booting_linux.md

Latest commit

 

History

History
201 lines (152 loc) · 11.1 KB

svm_booting_linux.md

File metadata and controls

201 lines (152 loc) · 11.1 KB

Demo - Linux Booting in a SVM

To verify Linux booting in SVM, we need the following components:

  • SVM components (in guest VM)

    • Linux kernel
    • RAM Disk image
    • Device tree

  • HOST HLOS components

    • Linux kernel with Gunyah Hypervisor linux drivers
    • crosvm user space app as VMM
    • rootfs disk image containing all host related tools and SVM image files

Following sections describe details on the images. These are already generated into mounted volumes as part of the docker image building process. If any of the images need to be rebuilt, the same script build-docker-img.sh can be used to create the tools, it will only re-create any image that's missing. If the sources need to be retained, then rm -rf command in the build scripts can be commented off.

SVM components (in guest VM)

Linux kernel

The linux kernel image generated during docker build process to test qemu/aarch64 functionality should work for this purpose.

RAM Disk image

The RAM Disk image (based on busybox) generated during docker build process to test qemu/aarch64 functionality should work here as well.

Device tree

Device tree will be generated by crosvm (VMM) running in Host HLOS (PVM) based on the command line arguments passed while launching it. Resource Manager will patch the DT before its passed on to the SVM Linux kernel.

HOST HLOS components

Note: Following details are provided only as information if anything need to be changed for experimentation purposes. All these following images have already been generated during the build-docker-img.sh step. So if you do not wish to change anything, proceed to step Test SVM loading

Linux kernel with Gunyah Hypervisor linux drivers

The linux kernel image generated during docker build process already patches the linux kernel sources with Gunyah hypervisor linux drivers. These details can be examined in the provided script clone-linux.sh which is used to build linux image.enables to test qemu/aarch64 functionality should work for this

crosvm user space app as VMM

A VMM is required to interact with Gunyah hypervisor to create, configure and launch the SVM and execute any OS in it. For the demo, crosvm user space app is built to handle this task. The details can be found in the included scripts clone-crosvm.sh and build-crosvm.sh.

Since crosvm sources and build can be really huge, sources and build output are deleted after copying the output binary image. This output binary is included in the rootfs disk image which is built after.

NOTE: Either a newer version of the Docker or a newer ubuntu distro (22.04 and later) has been encountering the following error..!! Observation was that the build works on host machine but doesn't in docker environment or vice versa. Currently we do not have a workaround for this problem, but if an older version of ubuntu distro (20.04) is available should work fine.

Trying to pull gcr.io/crosvm-infra/crosvm_dev:r0040...
Error: parsing image configuration: Get "https://storage.googleapis.com/artifacts.crosvm-infra.appspot.com/containers/images/sha256:6b0c28f5c282c677cc50a927e2227e5d89209ae3c6fa15db10593f7837a644fb": x509: certificate signed by unknown authority

Crosvm generated here will have a dependency on libgcc_s.so.1, which will be generated later during the rootfs disk image generation.

rootfs disk image containing all host related tools and SVM image files

Since the SVM linux kernel/ramdisk images and other host HLOS components need significant amount of storage space, we cannot just use ramdisk for host HLOS running in PVM. Instead we have to prepare the extfs formatted rootfs disk image that can be mounted as virtio disk. This method will provide the storage space for host HLOS without consuming the valuable RAM space.

Since generation of rootfs image also consumes huge space to generate, source and build files will be deleted after the binaries are copied to reference folder. If needed to retain source/build files, comment off the rm -rf command in the build shell scripts.

At a high level, this process does the following:

  • Downloads the stock linaro rootfs extfs disk image (tiny initramfs to keep the size small)
  • Downloads the Linaro OpenEmbedded Reference Platform Build (OE RPB) repo and generates the disk image
    • This step is taken only for the purpose of getting libgcc_s.so.1 file which is dependency for crosvm
    • This step takes significant amount of time
    • A potential optimization to avoid this huge build burden is a TODO later, but provides a potential if any useful components from this build can be utilized
  • Prepare reference folder for the filesystem
  • Prepare an empty extfs disk image of required size
  • Copy all the required files into the rootfs extfs disk image
    • File tree from stock Linaro rootfs
    • All kernel .ko files from the kernel build
    • crosvm (VMM)
    • ramdisk, kernel image (for SVM)

After the rootfs reference folder is created at the path ~/mnt/workspace/rootfs/reference, any additional file changes can be performed there. To make a new rootfs image with this modified rootfs reference folder, perform the following operations.

delete the generated rootfs file

rm -f ~/mnt/workspace/rootfs/rootfs-extfs-disk.img

and then run the rootfs generating script

cd ~/mnt/workspace/rootfs
build-rootfs-img.sh

This will add the environment variable into ~/mnt/workspace/.wsp-env make sure to remove the duplicates and retain only the relevant ones.

Alternately build-docker-img.sh can be run on host machine again to generate this image.

If any modifications to the reference tree is done that exceeds the disk size created, or if more free disk space needed in the PVM, the disk size value in the script build-rootfs-img.sh can be adjusted to any other value (say 1G)

Before:

. ~/utils/bldextfs.sh -f ${WORKSPACE}/rootfs/reference -o ${WORKSPACE}/rootfs/rootfs-extfs-disk.img -s 800M

After:

. ~/utils/bldextfs.sh -f ${WORKSPACE}/rootfs/reference -o ${WORKSPACE}/rootfs/rootfs-extfs-disk.img -s 1G

Test SVM loading

Now all the components needed to test linux running in SVM are ready. Run qemu with the following commands, note that dtb need to be generated again (When any size changes that can potentially move around stuff in the memory map/dt).

Note that the rootfs disk image is mounted RW, so any changes made will be persistent from HLOS running in PVM. So even the command history or any changes to .rc files would be available. But then it also means if the image gets corrupted it needs to be re-generated as mentioned above from the reference folder. If reference folder is deleted to reclaim space, keep a backup copy of the rootfs-extfs-disk.img file in order to avoid download/build time and space consumption.

cd ~/mnt/workspace
run-qemu.sh dtb

After dtb is generated, run the command to launch qemu

run-qemu.sh

This should result into booting to much complete linux environment in PVM via systemd init. Ignore the errors shown (unless those are fatal errors or crashes), if things went well, then a login prompt and a shell prompt should be seen

.
.
[  OK  ] Started User Login Management.
[  OK  ] Reached target Multi-User System.
         Starting Record Runlevel Change in UTMP...

Reference-Platform-Build-X11 3.0+linaro qemuarm64 ttyAMA0

qemuarm64 login: root (automatic login)
.
.
root@qemuarm64:~#

Once on the prompt, verify if the hypervisor related nodes showup

cd /sys/firmware/devicetree/base/hypervisor
find .

Now launch crosvm using the following command to create a SVM and run Linux OS in it. A shell script has been provided at /usr/gunyah/svm.sh to make it easier to launch.

/usr/gunyah/crosvm --no-syslog run --disable-sandbox \
--serial=type=stdout,hardware=virtio-console,console,stdin,num=1 \
--serial=type=stdout,hardware=serial,earlycon,num=1 \
--initrd /usr/gunyah/initrd.img  --no-balloon --no-rng \
--params "rw root=/dev/ram rdinit=/sbin/init earlyprintk=serial panic=0" \
/usr/gunyah/Image

OR

/usr/gunyah/svm.sh

The following shows the console logs of SVM. This could be pretty slow since each uart IO is going through the crosvm virtio-console (ie the control needs to switch from SVM back to PVM crosvm app to output the char to the actual console).

root@qemuarm64:~# /usr/gunyah/crosvm --no-syslog run --disable-sandbox --serial=type=stdout,hardware=virtio-console,console,stdin,num=1
--serial=type=stdout,hardware=serial,earlycon,num=1 --initrd /usr/gunyah/initrd.img  --no-balloon --no-rng --params "rw root=/dev/ram rd
init=/sbin/init earlyprintk=serial panic=0" /usr/gunyah/Image
[2023-09-15T21:44:00.904189296+00:00 INFO  crosvm] crosvm started.
[2023-09-15T21:44:00.935025408+00:00 INFO  crosvm] CLI arguments parsed.
[HYP] Emulated RAZ for ID register: ISS 0x36002f
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0480]
[    0.000000] Linux version 6.5.0-00025-gd9af2a6a5a22 (yvasi@buildkitsandbox) (clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a), LLD 15.0.7) #1 SMP PREEMPT Fri Sep 15 10:13:26 PDT 2023
[    0.000000] KASLR enabled
[    0.000000] random: crng init done
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] earlycon: uart8250 at MMIO 0x00000000000003f8 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x8ff709c0-0x8ff72fff]
[    0.000000] Zone ranges:

This script svm.sh can take additional arguments and pass on to crosvm app. As an example, we can provide the number of cpu's as additional argument to create SVM with more CPU cores. Note that adding more cores to SVM linux session might make console logs even slower..!!

/usr/gunyah/svm.sh --cpus 4

This should show that the Linux kernel is running in the SVM. Once verification of this linux environment is complete, issue poweroff command in the SVM linux shell to go back into the PVM linux environment. SVM can be launched again to verify a different configuration.

/ # poweroff
/ # umount: can't unmount /: Invalid argument
swapoff: can't open '/etc/fstab': No such file or directory
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[ 7204.608721] reboot: Power down

Note: Current latset version of Gunyah freezes on second time launch after SVM power off. This is working fine in the original version of gunyah before the updates, SVM relaunch feature can be verified on that original version of Gunyah/RM sources.

Press ctrl-a and x to exit Qemu

Follow-up TODO's:

The following areas require work, any help on these is appreciated:

  • Enable ssh_server in PVM Linux OS image, so that we can connect multiple ssh sessions and test multiple SVM sessions simultaneously.
  • Fix poor performance of the SVM Linux console, for example using the Gunayh RM Console driver.