Improving libvirt support for bhyve

• Student: Fabian Freyer (fabs@FreeBSD.org)
• Mentor: Roman Bogorodskiy (novel@FreeBSD.org)

The primary aim of this project is to implement missing calls and functionality in the libvirt bhyve driver. According to the libvirt API Support Matrix, there are a large number of calls not yet implemented. While some missing API calls are not applicable to bhyve, a number of them are, and these are aimed to be implemented in the scope of the project.

/* == Approach to solving the problem == */

/* (Details as to how you intend to approach the problem, if applicable. May include ideas as to how to break the problem down into smaller parts, or notes on research you need to do before settling on your final approach to solving the problem) */

This is a list of ideas of implementable features, however it is by no means complete, nor is the goal to implement all of these. It should rather be considered a braindump of possibilities:

• virConnectDomainXMLFromNative: This would mostly be an argument parser for a bhyve(8) and bhyvectl(8) command line.
• virConnectGetCPUModelNames: This needs some research: bhyve is not very flexible in configuration of what CPU model is exposed to the guest, need to figure out if that’s worth implementing now.

Most of these include some form of authentication handling and are therefore not applicable. However, the following do apply to bhyve and are easy to implement:

• virDomainGetMaxMemory: Since bhyve does not support memory ballooning, just return the amount of memory allocated here
• virDomainGetMaxVcpus, virDomainGetVcpus: I would use the approach described in this mailing list thread: “Until the vCPU state is exposed by bhyvectl, or we provide a sysctl, you can use heuristics: the number of vCPU threads for the bhyve process, or scan all vCPUs and only count those that have a non-zero RIP.”
• virDomainGetCPUStats: There are two ways to implement this, depending on whether usage should include or exclude the hypervisor time. If usage should include the hypervisor time, use standard CPU usage reporting as used by top et al. to determine CPU usage. If usage should exclude hypervisor time and include only vCPU time, use bhyvectl --get-stats to get the vcpu total runtime, and monitor this. I would have to do some research as to what to set this total runtime in relation to, and how to best monitor it.
• virDomainInjectNMI: Call bhyvectl --inject-nmi
• virDomainReset: Reset a bhyve VM with bhyvectl --force-reset and then clean things up using bhyvectl --destroy; update bhyve monitor code to handle exit code 0 from bhyve(8) that’s corresponding to reset (0 - reset / reboot, 1 - shutdown, 2 - halt) to trigger re-starting of the VM.
• virDomainReboot: Reboot a bhyve VM. Since bhyve does not support any facilities to trigger an ACPI G2 state by killing the domain with SIGTERM, then polling for the domain to shut down and restart it afterwards. This would be a rather hackish solution, so I’d like to hear feedback on this. However, the QEMU driver does it this way too, so I'm guessing that will be ok.

These would implement access to the vdev block storage layer. I plan to implement support for both file-backed and zvol-backed virtual machines for the following API calls:

• virDomainGetBlockInfo
• virDomainBlockPeek
• virDomainBlockCopy
• virDomainBlockStats
• virDomainBlockStatsFlags

Going further, since zvols support snapshotting, I plan to implement the following for zvol-backed storage

• virDomainBlockCommit
• virDomainBlockPull

This is blocked by getting virtfs-9p merged upstream I would like to create patches to support specifying filesystems when creating the domain as well as the following calls to be merged at a later time when VirtFS-9p support for bhyve becomes ready:

• virDomainFSFreeze
• virDomainFSThaw

• virDomainMemoryPeek: A guest’s memory space is exposed in /dev/vmm, so this call would have to read from that.
• Virt-host-validate(1) bhyve support: Add bhyve support to virt-host-validate
  • Check support of CPU supports features required for bhyve
  • Check if the vmm module is available
  • Check if essential networking things are available (if_bridge(4), if_tap(4)) + nmdm(4) for console

Currently the bhyve’s libvirt driver (as well as libxl/FreeBSD driver) only supports L2 interface bridging. There’s no support for upper level schemas like NAT for example. This is a huge task that involves research of what firefall is more applicable (ipfw or pf), designing of the firewall rules and the actual implemtation

I.e. the bhyve ... -s 7:0,passthru,4/0/0 thing. Probably that will involve the HAL nodedev driver modifications.

22 April - 23 May

• Improve ideas list
• Build a testing environment
• Read HACKING and look through the K&R style guide again
• Write simple patches to get accustomed to the patch submitting process
• Get an overview of the codebase

23 May - 20 June

In this period I aim to better familiarize myself with the codebase and get a better overview of what features are realistic to implement within the scope of GSoC.

Goals for this period are:

• Implement basic calls
• Implement virConnectDomainXMLFromNative
• Implement domain reset
• Start work implementing support booting via UEFI. This will entail:
  • Introducing a new capability checking whether BHYVE supports -l bootrom.
  • Researching the correct way to define the bootrom in the definition XML. At the moment several possibilities come to mind:
    • Checking whether the file specified in the <bootloader> tag is executable or not (if not executable, use it as a bootrom)
    • Specifying the firmware blob in a <loader type="pflash"> tag within the <os> tag.
  • Implementing the feature in virConnectDomainXMLToNative - especially omitting the loader command when a boot rom is specified needs some research if cases exist where bhyveload or grub-bhyve have to be called while booting using UEFI.
  • Implementing the feature in virConnectDomainXMLFromNative
• Research other features and define a plan for remaining time

27 June - 15 August

• Finish work on virConnectDomainXMLFromNative
• Implement virConnectGetDomainCapabilities
• Finish work on UEFI support
  • Update domain capabilities - here we could iterate over a list of UEFI firmwares, should they be installed on the system. Since it isn't really sure where they would be installed, do some research.
  • Probably creating a port containing uefi firmwares would be a good idea.
• Implement domain reboot and reset
• Implement inject-nmi
• Implement VNC display support for bhyve_graphics
  • Upstream support for VNC auth RFC 6143 7.1-7.2
  • Update domain capabilities

15 August - 23 August

Where possible, tests will be added to the libvirt tests.

Some testing ideas:

• virConnectDomainXMLFromNative: Create a round trip converting XML to native and then back, as well as native to xml and back. This isn't really practicable since we lose information.

Tests have been implemented and/or updated for:

• virConnectDomainXMLFromNative
• virConnectDomainXMLToNative:
  • bootrom support

I have a Fork on GitHub of the libvirt project, where I am pushing my code.

The following branches are currently used

libvirt:

• gsoc/bhyve_lpc_bootrom: implement support for booting via UEFI firmware. Posted to Mailinglist: v1
• gsoc/bhyve_gop: implement support for VNC framebuffer
• gsoc/fix/bhyveCapsBuild: fix emulator binary in virBhyveCapsBuild

virt-manager

• gsoc/bhyve-fixes

Ports:

• D7223/#211154: allow USE_GCC dependencies to be specified individually for {BUILD_,RUN_,TEST_}DEPENDS.
• D7225/#211157: metaport for bhyve firmware

I have also created a patch to bhyve_graphics:

• D7029

• 126e630 virConnectGetType: Trivially return "BHYVE"
• a1efc94 virConnectIsAlive: Trivially return 1, since “A connection will be classed as alive if it is […] local” and /dev/vmm is local.
• 32aa9ed virConnectIsEncrypted: Trivially return 0, since bhyve does not support encrypted interfaces.
• ef45eb9 virConnectIsSecure: Trivially return 1, since /dev/vmm is considered a “local connection”.
• b436a8a to dd23c38 gsoc/bhyveConnectDomainXMLFromNative Posted to Mailinglist: v1 v2 v3 v4 v5
• 32916b1 virConnectGetDomainCapabilities Posted to Mailinglist: v1

• 843a70a gsoc/fix/test-wrap-argv.pl Posted to Mailinglist: v1

• sysutils/uefi-edk2-bhyve
• sysutils/uefi-edk2-bhyve-csm

The following contributions are related to my work:

• [bhyve [patch print list of supported PCI and LPC devices by Roman Bogorodskiy
• D6999: Make kern.argmax boot tunable fixes make syntax-check on FreeBSD.