Porting bhyve to ARM-based platforms

• Student: Mihai Carabas (mihai@freebsd.org)
• Mentor: Peter Grehan (grehan@freebsd.org)

Nowadays virtualization is an important mechanism being approached by both, hardware designers and software developers. Given this necessity, the BSD community created a new hypervisor supporting hardware assisted virtualization. It is known under the name of bhyve. The current version of bhyve supports only the x86-based platforms (Intel and AMD, both having support for hardware assisted virtualization). In the last two years ARM has made significant progress developing virtualization extensions, this being implemented with success on other well-known hypervisors (KVM and Xen). Given the fact that ARM is the leader in embedded devices market and they target on long-term the server market, it will be a necessity for bhyve supporting ARM virtualization extensions. This projects proposes to implement support for hardware assisted virtualization on ARM platforms. It is not a trivial porting project due to the fact that ARM virtualization extensions doesn't allow access to normal CPU operation, like x86-based architectures.

In general, virtualization extensions adds a new privilege level for being able to isolate different guest OSes. On x86-based architectures this new privilege level comes with a new set of instructions in order to manage multiple guests. At the same time, the hypervisor has access to all other normal instructions, as running in supervisor mode (kernel mode). This scenario is the best fit for a hosted hypervisor (Type-2) [10], as bhyve is. In a Type-2 hypervisor both, guest OSes and normal processes, can run at the same time. The new level of privilege in ARM-based architectures it's called HYP-MODE. Compared to the x86 architecture, this mode only enables us the virtualization extensions instructions and we don't have access to the normal supervisor-mode ones. Running a Type-2 hypervisor in HYP-MODE isn't possible. The HYP-MODE is a best fit for Type-1 (baremetal) hypervisors like XEN [8] (it can run only virtual machines). Not being able to run a hosted hypervisor in HYP-MODE brings an important drawback: one cannot use the mechanisms (e.g. like memory management) already in-place in a hosted hypervisor and need to implement new ones.

bhyve was originally developed only for 64bit x86 platforms due to the fact that 32bit processor are too old and doesn't have virtualization support (or if they have its incomplete and not widespread at all). When talking about ARM platforms, the story changes: all the market devices have only 32bit processors. The 64bit ARM architecture was released by ARM but there isn't any development board supporting 64bit processors (only some emulators: Qemu and VExpress from ARM). Also FreeBSD support for 64bit is under heavy development. However ARM released the Virtualization Extensions on 32bit processors and this will have aproximatively full compatibility with the 64bit platforms. We will go developing bhyve support for ARM on a 32bit platform (see bellow).

Problems that will need to be tackled in order to be able to run an unmodified GuestOS on top of bhyve using an ARM platform: - configure u-boot bootloader to remain in hyp-mode, not drop to svc. - cpu virtualization: write the code that switches between the highvisor and guest OS - memory virtualization: this is similar to x86 memory virtualization with a difference: the HYP-mode lowvisor needs to maintain its own pagetable being in another address space than the highvisor - interrupt virtualization: expose a vGIC for each GuestOS - timer virtualization: ARM offers support for this, but these are likely to be shared between the hypervisor and the Guest OS, thus the configuration by the GuestOS will trap in the hypervisor in order to ensure isolation.

Week 1

- prepare FreeBSD ARM testing infrastructure (toolchains, boot-loader, the steps that need to be done to boot FreeBSD on the CubieBoard2)

- prepare an image to run KVM on CubieBoard2. This would be useful when we will want to see the instruction flow of different initializations (e.g. if we have a bug in early stages try to reproduce that exact flow on KVM to see what happens)

Week 2

- prepare the bootloader to run in HYP-MODE

- document about first instructions that need to be written in order to minimally run the FreeBSD hypervisor after running in HYP-MODE.

- document on the instructions that needs to be used to switch between the lowvisor and the highvisor

- document on the minimum requirements for lowvisor address space (build a special pagetable for it). Tackle the difference between normal ARM pagetables and the hypervisor ones (these are different, just like the EPT on Intel are different from the normal ones)

Week 3-5

- develop the logic from Week 2 to add support for some minimal traps caused by a Guest OS. We will start by executing some simple instruction from the GuestOS memory (not directly with a fully featured GuestOS)

- study the bhyve/bhyvectl user-space program and asses the modifications that need to be done in order to run a FreeBSD GuestOS image.

MIDTERM (week 6)

- memory virtualization (the pagetable for the lowvisor). We won't tackle advanced features like paging (we will hardwire the pagetable of the guest).

- CPU virtualization - support for minimal exceptions caused by a GuestOS

Week 7-9

- continue with implementing remaining traps necessary to run a GuestOS

- solve issues with memory virtualization for the GuestOS (dispatch pagefaults to the highvisor)

- probably new modifications in bhyve/bhyvectl will be needed

- document on how to deliver interrupts (timer interrupts) to a GuestOS

Week 10-11

- implement a mechanism for interrupt handling (solve traps caused by the GuestOS that is trying to configure the GIC)

FINAL (week 12)

- a set of patches that enables bhyve to run unmodified guests on ARM platforms

- a final report with the current status of the porting process

- due to the complexity of the project, 100% functionality cannot be ensured

Every week I will send an informative mail with my progress and the problems I had (see DragonflyBSD and FreeBSD reports from the previews summer of code years).

• building Bhyve test infrastructure
• find some suitable tests that generates a lot of interrupts and does not rely on any particular driver of a device (to not influence the tests)
• start implementing the custom hash-table that has as a key the instruction pointer (RIP), address of the “struct *vm” and the CR3 value (this basically identifies the address space for which we made the caching)

• write unit tests for the custom hash-table, especially for corner cases not to introduce critical bugs in kernel
• cache instructions by adding them to the hash-table. This would be done in vm_handle_inst_emul function [17]
• start looking where I can write-protect the page with the instruction and where I can handle the fault
• make some tests (may be performance) to see if there is any improvement. Of course there is a big chance to crash the guest

• functional code with some dummy instruction caching tests

• add protection to detect when the page tables modify by write protecting all the page table pages walked when we translate the instruction pointer(RIP) to the guest physical address
• make some stress tests to see if there is any corner case we've missed

• tests the solution on other Guest OSes
• centralize the results
• clean the code

• a set of patches implementing instruction caching for FreeBSD/VMM
• a final report with the results shown the improved/not improved results.

This kind of project is not suitable for a generic test plan (like unit tests, or action-based tests, etc). The real final goal of this project is running a FreeBSD GuestOS (a virtual machine) on top of bhyve using a CubieBoard2 platform. While coding different mechanisms of the virtualization infrastructure, we will verify the correctness by printing the state of the system at each important step. The progress will be given by the state of the GuestOS (how far it can boot).

• TODO

• TODO

• [1] http://svnweb.freebsd.org/base/stable/10/sys/amd64/vmm/vmm_instruction_emul.c?revision=262349&view=markup#l478
• [2] http://people.freebsd.org/~neel/bhyve/instruction_caching.txt
• [3] http://svnweb.freebsd.org/base/stable/10/sys/amd64/include/vmm_instruction_emul.h?revision=259073&view=markup#l44