Porting bhyve to ARM-based platforms

• Student: Mihai Carabas (mihai@freebsd.org)
• Mentor: Peter Grehan (grehan@freebsd.org)

Nowadays virtualization is an important mechanism being approached by both, hardware designers and software developers. Given this necessity, the BSD community created a new hypervisor supporting hardware assisted virtualization. It is known under the name of bhyve. The current version of bhyve supports only the x86-based platforms (Intel and AMD, both having support for hardware assisted virtualization). In the last two years ARM has made significant progress developing virtualization extensions, this being implemented with success on other well-known hypervisors (KVM and Xen). Given the fact that ARM is the leader in embedded devices market and they target on long-term the server market, it will be a necessity for bhyve supporting ARM virtualization extensions. This projects proposes to implement support for hardware assisted virtualization on ARM platforms. It is not a trivial porting project due to the fact that ARM virtualization extensions doesn't allow access to normal CPU operation, like x86-based architectures.

In general, virtualization extensions adds a new privilege level for being able to isolate different guest OSes. On x86-based architectures this new privilege level comes with a new set of instructions in order to manage multiple guests. At the same time, the hypervisor has access to all other normal instructions, as running in supervisor mode (kernel mode). This scenario is the best fit for a hosted hypervisor (Type-2) [1], as bhyve is. In a Type-2 hypervisor both, guest OSes and normal processes, can run at the same time. The new level of privilege in ARM-based architectures it's called HYP-MODE. Compared to the x86 architecture, this mode only enables us the virtualization extensions instructions and we don't have access to the normal supervisor-mode ones. Running a Type-2 hypervisor in HYP-MODE isn't possible. The HYP-MODE is a best fit for Type-1 (baremetal) hypervisors like XEN [2] (it can run only virtual machines). Not being able to run a hosted hypervisor in HYP-MODE brings an important drawback: one cannot use the mechanisms (e.g. like memory management) already in-place in a hosted hypervisor and need to implement new ones.

bhyve was originally developed only for 64bit x86 platforms due to the fact that 32bit processor are too old and doesn't have virtualization support (or if they have its incomplete and not widespread at all). When talking about ARM platforms, the story changes: all the market devices have only 32bit processors. The 64bit ARM architecture was released by ARM but there isn't any development board supporting 64bit processors (only some emulators: Qemu and VExpress from ARM). Also FreeBSD support for 64bit is under heavy development. However ARM released the Virtualization Extensions on 32bit processors and this will have aproximatively full compatibility with the 64bit platforms. We will go developing bhyve support for ARM on a 32bit platform (see bellow).

Problems that will need to be tackled in order to be able to run an unmodified GuestOS on top of bhyve using an ARM platform: - configure u-boot bootloader to remain in hyp-mode, not drop to svc. - cpu virtualization: write the code that switches between the highvisor and guest OS - memory virtualization: this is similar to x86 memory virtualization with a difference: the HYP-mode lowvisor needs to maintain its own pagetable being in another address space than the highvisor - interrupt virtualization: expose a vGIC for each GuestOS - timer virtualization: ARM offers support for this, but these are likely to be shared between the hypervisor and the Guest OS, thus the configuration by the GuestOS will trap in the hypervisor in order to ensure isolation.

• prepare FreeBSD ARM testing infrastructure (toolchains, boot-loader, the steps that need to be done to boot FreeBSD on the CubieBoard2)
• prepare an image to run KVM on CubieBoard2. This would be useful when we will want to see the instruction flow of different initializations (e.g. if we have a bug in early stages try to reproduce that exact flow on KVM to see what happens)

• prepare the bootloader to run in HYP-MODE
• document about first instructions that need to be written in order to minimally run the FreeBSD hypervisor after running in HYP-MODE.
• document on the instructions that needs to be used to switch between the lowvisor and the highvisor
• document on the minimum requirements for lowvisor address space (build a special pagetable for it). Tackle the difference between normal ARM pagetables and the hypervisor ones (these are different, just like the EPT on Intel are different from the normal ones)

• develop the logic from Week 2 to add support for some minimal traps caused by a Guest OS. We will start by executing some simple instruction from the GuestOS memory (not directly with a fully featured GuestOS)
• study the bhyve/bhyvectl user-space program and asses the modifications that need to be done in order to run a FreeBSD GuestOS image.

• memory virtualization (the pagetable for the lowvisor). We won't tackle advanced features like paging (we will hardwire the pagetable of the guest).
• CPU virtualization - support for minimal exceptions caused by a GuestOS

• continue with implementing remaining traps necessary to run a GuestOS
• solve issues with memory virtualization for the GuestOS (dispatch pagefaults to the highvisor)
• probably new modifications in bhyve/bhyvectl will be needed
• document on how to deliver interrupts (timer interrupts) to a GuestOS

• implement a mechanism for interrupt handling (solve traps caused by the GuestOS that is trying to configure the GIC)

• a set of patches that enables bhyve to run unmodified guests on ARM platforms
• a final report with the current status of the porting process
• due to the complexity of the project, 100% functionality cannot be ensured

This kind of project is not suitable for a generic test plan (like unit tests, or action-based tests, etc). The real final goal of this project is running a FreeBSD GuestOS (a virtual machine) on top of bhyve using a CubieBoard2 platform. While coding different mechanisms of the virtualization infrastructure, we will verify the correctness by printing the state of the system at each important step. The progress will be given by the state of the GuestOS (how far it can boot).

• Booting log FreeBSD on FVP_Cortex-A15 FastModel
• All the VMM code was duplicated in arch/arm/vmm and cleared-up of all x86_64 specific implementation. Basically only the stubs remained.
• Implemented the low-level context switch between the Host and Hyp-Mode in arch/arm/vmm/hyp.S
• Implemented LPAE page-tables because Hyp-Mode and Stage-2 translations on ARM32 support only LPAE (which isn't supported by the original FreeBSD PMAP)
• The bhyve, libvmmapi and bhyveload code was replicated and cleared-up remaing only the interfaces.
• Configured bhyveloadarm to allocate guest memory by hardwiring it (because we cannot rely on PMAP), load a kernel binary in memory at a preset address and make some runtime configurations (e.g. , KERNEL address, PERIPHBASE address)
• Configured bhyvearm to kick a created VM with bhyveloadarm. Added debug prints in key-points of the running path
• Implemented in the bhyvearm the emulation for MMIO acceses in userspace.
• Based on MMIO emulation, I've modified bvmconsole to work on ARM and was able to boot a guest that starting printing on the standard output.
• I've created the virtual GIC (VGIC) infrastructure which is adding a vGIC to a VM and is doing all the necessary mappings for CPU virtual interface for the VM and CPU virtual control interface for the Hypervisor.
• Final status: the guest boots up, it passes the GIC and timer initialization but ends up with “spurious interrupts” due to the lack of vGIC's distributor emulation

• https://socsvn.freebsd.org/socsvn/soc2015/mihai/
• mihai.micosconstruct.ro/bsdcam2015.pdf
• Steps to run FreeBSD in FastModels emulator:

- go at ARM site: https://silver.arm.com/browse/FM000 (we have to create an account)
- download Cortex-A15 VE FVP Package (Windows or Linux - depending on what you have)
- Install it and than run it like this:
"C:\Program Files (x86)\FVP_VE_Cortex-A15x124\models\Win64_VC2008\FVP_VE_Cortex-A15x1.exe" -f C:\gsoc2015\app.txt  C:\gsoc2015\linux-system-semi.axf
 
- The "linux-system-semi.axf" is obtained by compiling the boot-wrapper from my repo (you have it attached to this e-mail too):
gmake CROSS_COMPILE=arm-none-eabi- semi

- The app.txt contains:
""""
motherboard.smsc_91c111.enabled=1
motherboard.hostbridge.userNetworking=1
motherboard.pl011_uart0.out_file="uart.log"
cluster.cpu0.semihosting-cmd_line="--kernel C:\gsoc2015\kernel.bin -- "
""""
The --kernel parameter indicates the kernel.bin of the FreeBSD.

- In Windows, you must install the Telnet client: https://technet.microsoft.com/en-us/library/cc771275(v=ws.10).aspx

When running the FVP_VE_Cortex-A15x1.exe command, theoretically a console windows would appear. To be able to Debug (stepping and register view), you must install the "Model Debugger" which is included in the "Fast Models Evaluation". Here you must download a license.dat (based on your MAC address) and put it in the ARMLMD_LICENSE_FILE environment variable. It's a little tricky. If you have any trouble, don't hesitate to contact me.

After running the FVP_VE_Cortex-A15x1.exe and connect to it using the Model Debugger, hit the "Run" button to start simulation. After this a telnet console will appear. In there load the "vmm-arm" module:
"""
# kldload vmm-arm
arm_init hyp_code_start: 0xc5a07000, phys_hyp_code_start: 0x8c507000
l1pd = 8c514000
         2: l2pd = 8c519003
                 98: l3pd = 8c51b003
                         263: l3_entry = 8c507443
         3: l2pd = 8c516003
                 45: l3pd = 8c518003
                         0: l3_entry = 8c500443
                         7: l3_entry = 8c507443
arm_init init_hyp_vector: 0xc5a07700
arm_init hyp_l1pd: 0xc5a14000, phys_hyp_l1pd 0x8c514000
lpae_vmmmap_set n: 4096 4096
"""

Than load the guest image:
"""
# bhyveloadarm -k kernel.bin test
lpae_vmmmap_set n: 4096 13760
lpae_vmmmap_set n: 4096 9664
lpae_vmmmap_set n: 4096 5568
lpae_vmmmap_set n: 4096 1472
lpae_vmmmap_set n: 4096 4096
"""

Than run the virtual machine:
"""
# bhyvearm -b test
initarm: console initialized
 arg1 kmdp = 0xc10766d4
 boothowto = 0x00000000
 dtbp = 0xc0425288
 lastaddr1: 0xc1088000
loader passed (static) kenv:
 no env, null ptr
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #35 r288062:288801M: Fri Jul 24 13:40:07 EEST 2015
    root@freebsd64:/root/obj/arm.armv6/root/soc2015/mihai/bhyve-on-arm-head/sys/
FVP_VE_CORTEX_A15x1_GUEST arm
FreeBSD clang version 3.6.1 (tags/RELEASE_361/final 237755) 20150525
subsystem 1000000
   0xc0326518(0)... vm_mem_init:139
vm_mem_init:142
vm_mem_init:145
vm_mem_init:151
vm_mem_init:154
vm_mem_init:157
vm_mem_init:160
vm_mem_init:173
done.
   0xc033b438(0)... done.
subsystem 1800000
   0xc0102920(0)... done.
   0xc00dd4c0(0)... done.
   0xc00dceac(0xc1027900)... done.
   0xc00dceac(0xc10279ac)... done.
   0xc00dceac(0xc1028414)... done.
   0xc00dceac(0xc1028424)... done.
   0xc00dceac(0xc1028434)... done.
   0xc00dceac(0xc1028444)... done.
   0xc00dceac(0xc1028454)... done.
   0xc00dceac(0xc1028464)... done.
   0xc00dceac(0xc1028474)... done.
   0xc00dceac(0xc1028484)...
.....
subsystem 2100000
   0xc035503c(0)... CPU: Cortex A15-r2 rev 0 (Cortex-A core)
 Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext
 WB disabled EABT branch prediction enabled
LoUU:2 LoC:3 LoUIS:2
Cache level 1:
 32KB/64B 2-way data cache WB Read-Alloc Write-Alloc
 32KB/64B 2-way instruction cache Read-Alloc
Cache level 2:
 512KB/64B 16-way unified cache WB Read-Alloc Write-Alloc
real memory  = 134217728 (128 MB)
avail memory = 112324608 (107 MB)
done.
.....
gic0: <ARM Generic Interrupt Controller> mem 0x2c001000-0x2c001fff,0x2c002000-
0x2c003fff,0x2c004000-0x2c005fff,0x2c006000-0x2c007fff on ofwbus0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001000 size: 4 with val: 0
vgic_dist_mmio_read on cpu: 0 with gpa: 2c001004 size: 4
gic0: pn 0x390, arch 0x2, rev 0x0, implementer 0x43b irqs 32
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001400 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001800 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001404 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001804 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001408 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001808 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c00140c size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c00180c size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001410 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001810 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001414 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001814 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001418 size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001818 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c00141c size: 4 with val: 7fc
vgic_dist_mmio_write on cpu: 0 with gpa: 2c00181c size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001080 size: 4 with val: 0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001000 size: 4 with val: 1
timer0: <SP804 System Timer> mem 0x1c110000-0x1c110fff irq 34 on ofwbus0
vgic_dist_mmio_write on cpu: 0 with gpa: 2c001104 size: 4 with val: 4
Timecounter "SP804 Time Counter" frequency 1000000 Hz quality 1000
Event timer "SP804 Event Timer 0" frequency 62500 Hz quality 1000
timer0: peripheral ID: 00141804
timer0: PrimeCell ID: b105f00d
timer1: <SP804 System Timer> me
.....
timer1: PrimeCell ID: b105f00d
done.
   0xc036a030(0)... done.
   0xc034aaf8(0)... done.
Spurious interrupt detected
   0xc014ebb8(0)... done.
subsystem 4000000
Spurious interrupt detected
   0xc01a54Spurious interrupt detected
Spurious interrupt detected
30(0)... Spurious interrupt detected
Spurious interrupt detected
done.
Spurious interrupt detected
   module_register_init(0xc1029300)... done.
Spurious interrupt detected
   module_register_init(0xc1038120)... done.
Spurious interrupt detected
Spurious interrupt detected
   module_register_init(0xc10381b0)... done.
   0xc0192d58(0)... done.
   0xc019500c(0)... done.
   0xc018d0b4(0)... done.
Spurious interrupt detected
   module_register_init(0xc1029d50)... done.
   module_register_init(0xc1027300)... done.
Spurious interrupt detected
   module_register_init(0xc1028f0c)... done.
Spurious interrupt detected
pid 14 (init), uid 0: exited on signal 4
Spurious interrupt detected
Spurious interrupt detected
Spurious interrupt detected
"""

• [1] http://en.wikipedia.org/wiki/Hypervisor
• [2] http://www.xenproject.org/developers/teams/arm-hypervisor.html