summerofcode2016:improvinglibvirtsupportforbhyve

Improving libvirt support for bhyve

  • Student: Fabian Freyer (fabs@FreeBSD.org)
  • Mentor: Roman Bogorodskiy (novel@FreeBSD.org)

Project description

The primary aim of this project is to implement missing calls and functionality in the libvirt bhyve driver. According to the libvirt API Support Matrix, there are a large number of calls not yet implemented. While some missing API calls are not applicable to bhyve, a number of them are, and these are aimed to be implemented in the scope of the project.

/* == Approach to solving the problem == */

/* (Details as to how you intend to approach the problem, if applicable. May include ideas as to how to break the problem down into smaller parts, or notes on research you need to do before settling on your final approach to solving the problem) */

Deliverables

This is a list of ideas of implementable features, however it is by no means complete, nor is the goal to implement all of these. It should rather be considered a braindump of possibilities:

General calls

  • virConnectDomainXMLFromNative: This would mostly be an argument parser for a bhyve(8) and bhyvectl(8) command line.
  • virConnectGetCPUModelNames: This needs some research: bhyve is not very flexible in configuration of what CPU model is exposed to the guest, need to figure out if that’s worth implementing now.

Connection calls

Most of these include some form of authentication handling and are therefore not applicable. However, the following do apply to bhyve and are easy to implement:

General Domain calls

  • virDomainGetMaxMemory: Since bhyve does not support memory ballooning, just return the amount of memory allocated here
  • virDomainGetMaxVcpus, virDomainGetVcpus: I would use the approach described in this mailing list thread: “Until the vCPU state is exposed by bhyvectl, or we provide a sysctl, you can use heuristics: the number of vCPU threads for the bhyve process, or scan all vCPUs and only count those that have a non-zero RIP.”
  • virDomainGetCPUStats: There are two ways to implement this, depending on whether usage should include or exclude the hypervisor time. If usage should include the hypervisor time, use standard CPU usage reporting as used by top et al. to determine CPU usage. If usage should exclude hypervisor time and include only vCPU time, use bhyvectl --get-stats to get the vcpu total runtime, and monitor this. I would have to do some research as to what to set this total runtime in relation to, and how to best monitor it.
  • virDomainInjectNMI: Call bhyvectl --inject-nmi
  • virDomainReset: Reset a bhyve VM with bhyvectl --force-reset and then clean things up using bhyvectl --destroy; update bhyve monitor code to handle exit code 0 from bhyve(8) that’s corresponding to reset (0 - reset / reboot, 1 - shutdown, 2 - halt) to trigger re-starting of the VM.
  • virDomainReboot: Reboot a bhyve VM. Since bhyve does not support any facilities to trigger an ACPI G2 state by killing the domain with SIGTERM, then polling for the domain to shut down and restart it afterwards. This would be a rather hackish solution, so I’d like to hear feedback on this. However, the QEMU driver does it this way too, so I'm guessing that will be ok.

Block-Device level calls

These would implement access to the vdev block storage layer. I plan to implement support for both file-backed and zvol-backed virtual machines for the following API calls:

  • virDomainGetBlockInfo
  • virDomainBlockPeek
  • virDomainBlockCopy
  • virDomainBlockStats
  • virDomainBlockStatsFlags

Going further, since zvols support snapshotting, I plan to implement the following for zvol-backed storage

  • virDomainBlockCommit
  • virDomainBlockPull

VirtFS layer

This is blocked by getting virtfs-9p merged upstream I would like to create patches to support specifying filesystems when creating the domain as well as the following calls to be merged at a later time when VirtFS-9p support for bhyve becomes ready:

  • virDomainFSFreeze
  • virDomainFSThaw

Memory inspection

  • virDomainMemoryPeek: A guest’s memory space is exposed in /dev/vmm, so this call would have to read from that.
  • Virt-host-validate(1) bhyve support: Add bhyve support to virt-host-validate
    • Check support of CPU supports features required for bhyve
    • Check if the vmm module is available
    • Check if essential networking things are available (if_bridge(4), if_tap(4)) + nmdm(4) for console

Networking support

Currently the bhyve’s libvirt driver (as well as libxl/FreeBSD driver) only supports L2 interface bridging. There’s no support for upper level schemas like NAT for example. This is a huge task that involves research of what firefall is more applicable (ipfw or pf), designing of the firewall rules and the actual implemtation

PCI passthrough support

I.e. the bhyve ... -s 7:0,passthru,4/0/0 thing. Probably that will involve the HAL nodedev driver modifications.

Milestones

Communtiy Bonding Period

22 April - 23 May

  • Improve ideas list
  • Build a testing environment
  • Read HACKING and look through the K&R style guide again
  • Write simple patches to get accustomed to the patch submitting process
  • Get an overview of the codebase

First Work Period

23 May - 20 June

In this period I aim to better familiarize myself with the codebase and get a better overview of what features are realistic to implement within the scope of GSoC.

Goals for this period are:

  • Implement basic calls
  • Implement virConnectDomainXMLFromNative
  • Implement domain reset
  • Start work implementing support booting via UEFI. This will entail:
    • Introducing a new capability checking whether BHYVE supports -l bootrom.
    • Researching the correct way to define the bootrom in the definition XML. At the moment several possibilities come to mind:
      • Checking whether the file specified in the <bootloader> tag is executable or not (if not executable, use it as a bootrom)
      • Specifying the firmware blob in a <loader type="pflash"> tag within the <os> tag.
    • Implementing the feature in virConnectDomainXMLToNative - especially omitting the loader command when a boot rom is specified needs some research if cases exist where bhyveload or grub-bhyve have to be called while booting using UEFI.
    • Implementing the feature in virConnectDomainXMLFromNative
  • Research other features and define a plan for remaining time

Second Work Period

27 June - 15 August

Last Week

15 August - 23 August

Test Plan

Where possible, tests will be added to the libvirt tests.

Some testing ideas:

  • virConnectDomainXMLFromNative: Create a round trip converting XML to native and then back, as well as native to xml and back. This isn't really practicable since we lose information.

Tests have been implemented and/or updated for:

The Code

WIP

I have a Fork on GitHub of the libvirt project, where I am pushing my code.

The following branches are currently used

libvirt:

virt-manager

Ports:

  • D7223/#211154: allow USE_GCC dependencies to be specified individually for {BUILD_,RUN_,TEST_}DEPENDS.
  • D7225/#211157: metaport for bhyve firmware

I have also created a patch to bhyve_graphics:

Upstreamed

Connection Calls

Bugfixes

Ports

Useful links

The following contributions are related to my work:

  • summerofcode2016/improvinglibvirtsupportforbhyve.txt
  • Last modified: 2016/07/17 03:27
  • by FabianFreyer