Porting bhyve to ARM-based platforms

• Student: Mihai Carabas (mihai@freebsd.org)
• Mentor: Peter Grehan (grehan@freebsd.org)

Nowadays virtualization is an important mechanism being approached by both, hardware designers and software developers. Given this necessity, the BSD community created a new hypervisor supporting hardware assisted virtualization. It is known under the name of bhyve. The current version of bhyve supports only the x86-based platforms (Intel and AMD, both having support for hardware assisted virtualization). In the last two years ARM has made significant progress developing virtualization extensions, this being implemented with success on other well-known hypervisors (KVM and Xen). Given the fact that ARM is the leader in embedded devices market and they target on long-term the server market, it will be a necessity for bhyve supporting ARM virtualization extensions. This projects proposes to implement support for hardware assisted virtualization on ARM platforms. It is not a trivial porting project due to the fact that ARM virtualization extensions doesn't allow access to normal CPU operation, like x86-based architectures.

In general, virtualization extensions adds a new privilege level for being able to isolate different guest OSes. On x86-based architectures this new privilege level comes with a new set of instructions in order to manage multiple guests. At the same time, the hypervisor has access to all other normal instructions, as running in supervisor mode (kernel mode). This scenario is the best fit for a hosted hypervisor (Type-2) [1], as bhyve is. In a Type-2 hypervisor both, guest OSes and normal processes, can run at the same time. The new level of privilege in ARM-based architectures it's called HYP-MODE. Compared to the x86 architecture, this mode only enables us the virtualization extensions instructions and we don't have access to the normal supervisor-mode ones. Running a Type-2 hypervisor in HYP-MODE isn't possible. The HYP-MODE is a best fit for Type-1 (baremetal) hypervisors like XEN [2] (it can run only virtual machines). Not being able to run a hosted hypervisor in HYP-MODE brings an important drawback: one cannot use the mechanisms (e.g. like memory management) already in-place in a hosted hypervisor and need to implement new ones.

bhyve was originally developed only for 64bit x86 platforms due to the fact that 32bit processor are too old and doesn't have virtualization support (or if they have its incomplete and not widespread at all). When talking about ARM platforms, the story changes: all the market devices have only 32bit processors. The 64bit ARM architecture was released by ARM but there isn't any development board supporting 64bit processors (only some emulators: Qemu and VExpress from ARM). Also FreeBSD support for 64bit is under heavy development. However ARM released the Virtualization Extensions on 32bit processors and this will have aproximatively full compatibility with the 64bit platforms. We will go developing bhyve support for ARM on a 32bit platform (see bellow).

Problems that will need to be tackled in order to be able to run an unmodified GuestOS on top of bhyve using an ARM platform: - configure u-boot bootloader to remain in hyp-mode, not drop to svc. - cpu virtualization: write the code that switches between the highvisor and guest OS - memory virtualization: this is similar to x86 memory virtualization with a difference: the HYP-mode lowvisor needs to maintain its own pagetable being in another address space than the highvisor - interrupt virtualization: expose a vGIC for each GuestOS - timer virtualization: ARM offers support for this, but these are likely to be shared between the hypervisor and the Guest OS, thus the configuration by the GuestOS will trap in the hypervisor in order to ensure isolation.

• prepare FreeBSD ARM testing infrastructure (toolchains, boot-loader, the steps that need to be done to boot FreeBSD on the CubieBoard2)
• prepare an image to run KVM on CubieBoard2. This would be useful when we will want to see the instruction flow of different initializations (e.g. if we have a bug in early stages try to reproduce that exact flow on KVM to see what happens)

• prepare the bootloader to run in HYP-MODE
• document about first instructions that need to be written in order to minimally run the FreeBSD hypervisor after running in HYP-MODE.
• document on the instructions that needs to be used to switch between the lowvisor and the highvisor
• document on the minimum requirements for lowvisor address space (build a special pagetable for it). Tackle the difference between normal ARM pagetables and the hypervisor ones (these are different, just like the EPT on Intel are different from the normal ones)

• develop the logic from Week 2 to add support for some minimal traps caused by a Guest OS. We will start by executing some simple instruction from the GuestOS memory (not directly with a fully featured GuestOS)
• study the bhyve/bhyvectl user-space program and asses the modifications that need to be done in order to run a FreeBSD GuestOS image.

• memory virtualization (the pagetable for the lowvisor). We won't tackle advanced features like paging (we will hardwire the pagetable of the guest).
• CPU virtualization - support for minimal exceptions caused by a GuestOS

• continue with implementing remaining traps necessary to run a GuestOS
• solve issues with memory virtualization for the GuestOS (dispatch pagefaults to the highvisor)
• probably new modifications in bhyve/bhyvectl will be needed
• document on how to deliver interrupts (timer interrupts) to a GuestOS

• implement a mechanism for interrupt handling (solve traps caused by the GuestOS that is trying to configure the GIC)

• a set of patches that enables bhyve to run unmodified guests on ARM platforms
• a final report with the current status of the porting process
• due to the complexity of the project, 100% functionality cannot be ensured

This kind of project is not suitable for a generic test plan (like unit tests, or action-based tests, etc). The real final goal of this project is running a FreeBSD GuestOS (a virtual machine) on top of bhyve using a CubieBoard2 platform. While coding different mechanisms of the virtualization infrastructure, we will verify the correctness by printing the state of the system at each important step. The progress will be given by the state of the GuestOS (how far it can boot).

• Booting log FreeBSD on FVP_Cortex-A15 FastModel
• All the VMM code was duplicated in arch/arm/vmm and cleared-up of all x86_64 specific implementation. Basically only the stubs remained.
• Implemented the low-level context switch between the Host and Hyp-Mode in arch/arm/vmm/hyp.S
• Implemented LPAE page-tables because Hyp-Mode and Stage-2 translations on ARM32 support only LPAE (which isn't supported by the original FreeBSD PMAP)
• The bhyve, libvmmapi and bhyveload code was replicated and cleared-up remaing only the interfaces.
• Configured bhyveloadarm to allocate guest memory by hardwiring it (because we cannot rely on PMAP), load a kernel binary in memory at a preset address and make some runtime configurations (e.g. , KERNEL address, PERIPHBASE address)
• Configured bhyvearm to kick a created VM with bhyveloadarm. Added debug prints in key-points of the running path
• Implemented in the bhyvearm the emulation for MMIO acceses in userspace.
• Based on MMIO emulation, I've modified bvmconsole to work on ARM and was able to boot a guest that starting printing on the standard output.
• I've created the virtual GIC (VGIC) infrastructure which is adding a vGIC to a VM and is doing all the necessary mappings.
• Final status: the guest boots up, it passes the GIC and timer initialization but ends up with “spurious interrupts” due to the lack of vGIC's distributor emulation

• https://socsvn.freebsd.org/socsvn/soc2015/mihai/
• Steps to run FreeBSD in FastModels emulator:

- go at ARM site: https://silver.arm.com/browse/FM000 (we have to create an account) - download Cortex-A15 VE FVP Package (Windows or Linux - depending on what you have) - Install it and than run it like this: “C:\Program Files (x86)\FVP_VE_Cortex-A15x124\models\Win64_VC2008\FVP_VE_Cortex-A15x1.exe” -f C:\gsoc2015\app.txt C:\gsoc2015\linux-system-semi.axf

- The “linux-system-semi.axf” is obtained by compiling the boot-wrapper from my repo (you have it attached to this e-mail too): gmake CROSS_COMPILE=arm-none-eabi- semi

- The app.txt contains: “”“” motherboard.smsc_91c111.enabled=1 motherboard.hostbridge.userNetworking=1 motherboard.pl011_uart0.out_file=“uart.log” cluster.cpu0.semihosting-cmd_line=“–kernel C:\gsoc2015\kernel.bin – ” “”“” The –kernel parameter indicates the kernel.bin of the FreeBSD.

- In Windows, you must install the Telnet client: https://technet.microsoft.com/en-us/library/cc771275(v=ws.10).aspx

When running the FVP_VE_Cortex-A15x1.exe command, teoretically a console windows would appear. To be able to Debug (stepping and register view), you must install the “Model Debugger” which is included in the “Fast Models Evaluation”. Here you must download a license.dat (based on your MAC address) and put it in the ARMLMD_LICENSE_FILE environment variable. It's a little tricky.

If you have any trouble, don't hesitate to contact me.

• [1] http://en.wikipedia.org/wiki/Hypervisor
• [2] http://www.xenproject.org/developers/teams/arm-hypervisor.html