Porting bhyve to ARM-based platforms

This is an old revision of the document!

Student: Mihai Carabas (mihai@freebsd.org)
Mentor: Peter Grehan (grehan@freebsd.org)

Project description

Nowadays virtualization is an important mechanism being approached by both, hardware designers and software developers. Given this necessity, the BSD community created a new hypervisor supporting hardware assisted virtualization. It is known under the name of bhyve. The current version of bhyve supports only the x86-based platforms (Intel and AMD, both having support for hardware assisted virtualization). In the last two years ARM has made significant progress developing virtualization extensions, this being implemented with success on other well-known hypervisors (KVM and Xen). Given the fact that ARM is the leader in embedded devices market and they target on long-term the server market, it will be a necessity for bhyve supporting ARM virtualization extensions. This projects proposes to implement support for hardware assisted virtualization on ARM platforms. It is not a trivial porting project due to the fact that ARM virtualization extensions doesn't allow access to normal CPU operation, like x86-based architectures.

In general, virtualization extensions adds a new privilege level for being able to isolate different guest OSes. On x86-based architectures this new privilege level comes with a new set of instructions in order to manage multiple guests. At the same time, the hypervisor has access to all other normal instructions, as running in supervisor mode (kernel mode). This scenario is the best fit for a hosted hypervisor (Type-2) [10], as bhyve is. In a Type-2 hypervisor both, guest OSes and normal processes, can run at the same time. The new level of privilege in ARM-based architectures it's called HYP-MODE. Compared to the x86 architecture, this mode only enables us the virtualization extensions instructions and we don't have access to the normal supervisor-mode ones. Running a Type-2 hypervisor in HYP-MODE isn't possible. The HYP-MODE is a best fit for Type-1 (baremetal) hypervisors like XEN [8] (it can run only virtual machines). Not being able to run a hosted hypervisor in HYP-MODE brings an important drawback: one cannot use the mechanisms (e.g. like memory management) already in-place in a hosted hypervisor and need to implement new ones.

bhyve was originally developed only for 64bit x86 platforms due to the fact that 32bit processor are too old and doesn't have virtualization support (or if they have its incomplete and not widespread at all). When talking about ARM platforms, the story changes: all the market devices have only 32bit processors. The 64bit ARM architecture was released by ARM but there isn't any development board supporting 64bit processors (only some emulators: Qemu and VExpress from ARM). Also FreeBSD support for 64bit is under heavy development. However ARM released the Virtualization Extensions on 32bit processors and this will have aproximatively full compatibility with the 64bit platforms. We will go developing bhyve support for ARM on a 32bit platform (see bellow).

Problems that will need to be tackled in order to be able to run an unmodified GuestOS on top of bhyve using an ARM platform: - configure u-boot bootloader to remain in hyp-mode, not drop to svc. - cpu virtualization: write the code that switches between the highvisor and guest OS - memory virtualization: this is similar to x86 memory virtualization with a difference: the HYP-mode lowvisor needs to maintain its own pagetable being in another address space than the highvisor - interrupt virtualization: expose a vGIC for each GuestOS - timer virtualization: ARM offers support for this, but these are likely to be shared between the hypervisor and the Guest OS, thus the configuration by the GuestOS will trap in the hypervisor in order to ensure isolation.

Week 1

- prepare FreeBSD ARM testing infrastructure (toolchains, boot-loader, the steps that need to be done to boot FreeBSD on the CubieBoard2)

- prepare an image to run KVM on CubieBoard2. This would be useful when we will want to see the instruction flow of different initializations (e.g. if we have a bug in early stages try to reproduce that exact flow on KVM to see what happens)

Week 2

- prepare the bootloader to run in HYP-MODE

- document about first instructions that need to be written in order to minimally run the FreeBSD hypervisor after running in HYP-MODE.

- document on the instructions that needs to be used to switch between the lowvisor and the highvisor

- document on the minimum requirements for lowvisor address space (build a special pagetable for it). Tackle the difference between normal ARM pagetables and the hypervisor ones (these are different, just like the EPT on Intel are different from the normal ones)

Week 3-5

- develop the logic from Week 2 to add support for some minimal traps caused by a Guest OS. We will start by executing some simple instruction from the GuestOS memory (not directly with a fully featured GuestOS)

- study the bhyve/bhyvectl user-space program and asses the modifications that need to be done in order to run a FreeBSD GuestOS image.

MIDTERM (week 6)

- memory virtualization (the pagetable for the lowvisor). We won't tackle advanced features like paging (we will hardwire the pagetable of the guest).

- CPU virtualization - support for minimal exceptions caused by a GuestOS

Week 7-9

- continue with implementing remaining traps necessary to run a GuestOS

- solve issues with memory virtualization for the GuestOS (dispatch pagefaults to the highvisor)

- probably new modifications in bhyve/bhyvectl will be needed

- document on how to deliver interrupts (timer interrupts) to a GuestOS

Week 10-11

- implement a mechanism for interrupt handling (solve traps caused by the GuestOS that is trying to configure the GIC)

FINAL (week 12)

- a set of patches that enables bhyve to run unmodified guests on ARM platforms

- a final report with the current status of the porting process

- due to the complexity of the project, 100% functionality cannot be ensured

Every week I will send an informative mail with my progress and the problems I had (see DragonflyBSD and FreeBSD reports from the previews summer of code years).

building Bhyve test infrastructure
find some suitable tests that generates a lot of interrupts and does not rely on any particular driver of a device (to not influence the tests)
start implementing the custom hash-table that has as a key the instruction pointer (RIP), address of the “struct *vm” and the CR3 value (this basically identifies the address space for which we made the caching)

write unit tests for the custom hash-table, especially for corner cases not to introduce critical bugs in kernel
cache instructions by adding them to the hash-table. This would be done in vm_handle_inst_emul function [17]
start looking where I can write-protect the page with the instruction and where I can handle the fault
make some tests (may be performance) to see if there is any improvement. Of course there is a big chance to crash the guest

functional code with some dummy instruction caching tests

add protection to detect when the page tables modify by write protecting all the page table pages walked when we translate the instruction pointer(RIP) to the guest physical address
make some stress tests to see if there is any corner case we've missed

tests the solution on other Guest OSes
centralize the results
clean the code

a set of patches implementing instruction caching for FreeBSD/VMM
a final report with the results shown the improved/not improved results.

The real final goal of this project are the results obtained after implementing the instruction caching. Before start testing we should identify where is the overhead in the worse case scenario: trying to search the instruction in the hashtable and also fetch and decode the instruction (this would be worse than the current case). However as Neel mentioned in the FreeBSD case, the scenario will only take place at boot-time.

The tests that will be run will have to generate lot's of interrupts (causing VM exits which would need instruction decode) in order to see the specific difference with and without instruction decode. Here I guess I could steel two IBM blades to install FreeBSD on them in order to ease-up the testing (I don't know if a runtime config would work in this sensitive case).

Other tests scenario's will be using different GuestOS-es to see the gain we have in there.

First of all you can find all the steps I've taken through the project in the soc-status list, in the thread [GSOC] bhyve instruction caching .

Until now I managed to finish up all the coding stuff related to instruction caching. As you saw in my soc-status e-mails we obtained a speed-up of 35%-40% in the microbenchmarking tests (accessing LAPIC many times from a kernel module). Further we wanted to see how get this extrapolated to real-world workloads.

I've made two kinds of benchmarking: a CPU intensive process and a make buildworld -j2 command. For each of one I've measured the time spent to execute.

The CPU intensive application

The CPU intensive application is a bash script:

#!/usr/local/bin/bash
a=0
MAX=10000000
for i in $(seq 1 $MAX);
do
        a=$((a+1))
done

For a VM with 2 vCPUs:

Cache_instr=1

real    3m45.067s 3m42.628s 3m38.371s 3m36.301s 3m39.929s
user    3m10.454s 3m8.785s 3m7.516s 3m8.204s 3m8.822s
sys     0m19.085s 0m16.135s 0m13.696s 0m13.016s 0m16.105s

Cache_instru=0

real    3m50.550s 3m41.517s 3m34.783s
user    3m5.350s 3m7.571s 3m1.415s
sys     0m25.268s 0m19.200s 0m16.200s

There are multiple measurements. As you can see the results aren't stable and are in the same range. To minimize the range they vary, I repeated the tests with 1vCPU (to eliminate the context switches):

* Cache_instr=1

real    2m58.968s 2m57.009s 3m0.451s 2m55.902s 2m56.422s
user    2m46.909s 2m45.241s 2m45.670s 2m45.788s 2m45.503s
sys     0m4.890s 0m4.134s 0m3.942s 0m3.764s 0m3.984s

* Cache_instr=0

real    2m56.845s 2m57.051s 3m1.794s 2m57.340s
user    2m45.232s 2m44.873s 2m45.482s 2m46.538s
sys     0m4.644s 0m4.141s 0m3.906s 0m3.875s

As you can see the results are very appropiate in terms of variation and almost the same.

make buildworld -j2

The results for a make buildworld -j2 with 1 vCPU:

* Cache_instr=1

13900.60 real     12051.54 user      1800.42 sys

* Cache_instr=0

13938.07 real     12122.14 user      1743.61 sys

As you can see the difference between them is not significant and is about the same.

Also for this two different kind of workloads there is no speed-up improvement unfortunatelly.

I've tried other workloads more specific like:

dd if=/dev/zero of=/dev/zero bs=256 count=10000K (from memory to memory - to not be influenced by the storage system)
A simple getuid program that executes getuid syscall in a loop:

int main(int argc, char *argv[])
{
        int i;
        if (argc == 2) {
                i = atoi(argv[1]);
        } else {
                i = 100;
        }
        while (i > 0) {
                getuid();
                i--;
        }
        return 0;
}

The results were the same with and without instruction caching.

It seems that we can't get a real-world benefict with this instruction caching.