====== Improving libvirt support for bhyve ====== * Student: Fabian Freyer (fabs@%%FreeBSD%%.org) * Mentor: Roman Bogorodskiy (novel@%%FreeBSD%%.org) ===== Project description ===== The primary aim of this project is to implement missing calls and functionality in the libvirt bhyve driver. According to the libvirt [[https://libvirt.org/hvsupport.html|API Support Matrix]], there are a large number of calls not yet implemented. While some missing API calls are not applicable to bhyve, a number of them are, and these are aimed to be implemented in the scope of the project. /* == Approach to solving the problem == */ /* (Details as to how you intend to approach the problem, if applicable. May include ideas as to how to break the problem down into smaller parts, or notes on research you need to do before settling on your final approach to solving the problem) */ ===== Deliverables ===== This is a list of ideas of implementable features, however it is by no means complete, nor is the goal to implement all of these. It should rather be considered a braindump of possibilities: ==== General calls ==== * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]]: This would mostly be an argument parser for a [[https://www.freebsd.org/cgi/man.cgi?query=bhyve&manpath=FreeBSD+10.3-RELEASE+and+Ports|bhyve(8)]] and [[https://www.freebsd.org/cgi/man.cgi?query=bhyvectl&manpath=FreeBSD+10.3-RELEASE+and+Ports|bhyvectl(8)]] command line. * virConnectGetCPUModelNames: This needs some research: bhyve is not very flexible in configuration of what CPU model is exposed to the guest, need to figure out if that’s worth implementing now. ==== Connection calls ==== Most of these include some form of authentication handling and are therefore not applicable. However, the following do apply to bhyve and are easy to implement: ==== General Domain calls ==== * virDomainGetMaxMemory: Since bhyve does not support memory ballooning, just return the amount of memory allocated here * virDomainGetMaxVcpus, virDomainGetVcpus: I would use the approach described in [[https://lists.freebsd.org/pipermail/freebsd-virtualization/2014-April/002469.html|this mailing list thread]]: “Until the vCPU state is exposed by bhyvectl, or we provide a sysctl, you can use heuristics: the number of vCPU threads for the bhyve process, or scan all vCPUs and only count those that have a non-zero RIP.” * virDomainGetCPUStats: There are two ways to implement this, depending on whether usage should include or exclude the hypervisor time. If usage should include the hypervisor time, use standard CPU usage reporting as used by top et al. to determine CPU usage. If usage should exclude hypervisor time and include only vCPU time, use ''%%bhyvectl --get-stats%%'' to get the vcpu total runtime, and monitor this. I would have to do some research as to what to set this total runtime in relation to, and how to best monitor it. * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainInjectNMI|virDomainInjectNMI]]: Call ''%%bhyvectl --inject-nmi%%'' * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainReset|virDomainReset]]: Reset a bhyve VM with ''%%bhyvectl --force-reset%%'' and then clean things up using ''%%bhyvectl --destroy%%''; update bhyve monitor code to handle exit code 0 from bhyve(8) that’s corresponding to reset (0 - reset / reboot, 1 - shutdown, 2 - halt) to trigger re-starting of the VM. * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainReboot|virDomainReboot]]: Reboot a bhyve VM. Since bhyve does not support any facilities to trigger an ACPI G2 state by killing the domain with SIGTERM, then polling for the domain to shut down and restart it afterwards. This would be a rather hackish solution, so I’d like to hear feedback on this. However, the QEMU driver does it this way too, so I'm guessing that will be ok. ==== Block-Device level calls ==== These would implement access to the vdev block storage layer. I plan to implement support for both file-backed and zvol-backed virtual machines for the following API calls: * virDomainGetBlockInfo * virDomainBlockPeek * virDomainBlockCopy * virDomainBlockStats * virDomainBlockStatsFlags Going further, since zvols support snapshotting, I plan to implement the following for zvol-backed storage * virDomainBlockCommit * virDomainBlockPull ==== VirtFS layer ==== This is blocked by getting virtfs-9p merged upstream I would like to create patches to support specifying filesystems when creating the domain as well as the following calls to be merged at a later time when %%VirtFS%%-9p support for bhyve becomes ready: * virDomainFSFreeze * virDomainFSThaw ==== Memory inspection ==== * virDomainMemoryPeek: A guest’s memory space is exposed in ''%%/dev/vmm%%'', so this call would have to read from that. * [[https://www.freebsd.org/cgi/man.cgi?query=virt-host-validate&manpath=FreeBSD+10.3-RELEASE+and+Ports|Virt-host-validate(1)]] bhyve support: Add bhyve support to virt-host-validate * Check support of CPU supports features required for bhyve * Check if the ''%%vmm%%'' module is available * Check if essential networking things are available (if_bridge(4), if_tap(4%%))%% + nmdm(4) for console ==== Networking support ==== Currently the bhyve’s libvirt driver (as well as libxl/%%FreeBSD%% driver) only supports L2 interface bridging. There’s no support for upper level schemas like NAT for example. This is a huge task that involves research of what firefall is more applicable (ipfw or pf), designing of the firewall rules and the actual implemtation ==== PCI passthrough support ==== I.e. the ''%%bhyve ... -s 7:0,passthru,4/0/0%%'' thing. Probably that will involve the HAL nodedev driver modifications. ===== Milestones ===== ==== Communtiy Bonding Period ==== 22 April - 23 May * Improve ideas list * Build a testing environment * Read HACKING and look through the K&R style guide again * Write simple patches to get accustomed to the patch submitting process * Get an overview of the codebase ==== First Work Period ==== 23 May - 20 June In this period I aim to better familiarize myself with the codebase and get a better overview of what features are realistic to implement within the scope of %%GSoC%%. Goals for this period are: * Implement basic calls * Implement [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]] * Implement domain reset * Start work implementing support booting via [[bhyve/UEFI|UEFI]]. This will entail: * Introducing a new capability checking whether BHYVE supports ''%%-l bootrom%%''. * Researching the correct way to define the bootrom in the definition XML. At the moment several possibilities come to mind: * Checking whether the file specified in the %%<%%bootloader> tag is executable or not (if not executable, use it as a bootrom) * Specifying the firmware blob in a [[https://libvirt.org/formatdomain.html#elementsOSKernel|]] tag within the %%<%%os> tag. * Implementing the feature in virConnectDomainXMLToNative - especially omitting the loader command when a boot rom is specified needs some research if cases exist where bhyveload or grub-bhyve have to be called while booting using [[bhyve/UEFI|UEFI]]. * Implementing the feature in [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]] * Research other features and define a plan for remaining time ==== Second Work Period ==== 27 June - 15 August * Finish work on [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]] * Implement [[http://libvirt.org/formatdomaincaps.html|virConnectGetDomainCapabilities]] * Finish work on [[bhyve/UEFI|UEFI]] support * [[http://libvirt.org/formatdomaincaps.html#elementsOSBIOS|Update domain capabilities]] - here we could iterate over a list of UEFI firmwares, should they be installed on the system. Since it isn't really sure where they would be installed, do some research. * Probably creating a port containing uefi firmwares would be a good idea. * Implement domain reboot and reset * Implement inject-nmi * Implement VNC display support for [[https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-May/004471.html|bhyve_graphics]] * [[https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-June/004535.html|Upstream]] support for VNC auth [[https://tools.ietf.org/html/rfc6143|RFC 6143]] [[https://tools.ietf.org/html/rfc6143#section-7.1|7.1]]-[[https://tools.ietf.org/html/rfc6143#section-7.2|7.2]] * [[http://libvirt.org/formatdomaincaps.html#elementsGraphics|Update domain capabilities]] ==== Last Week ==== 15 August - 23 August ===== Test Plan ===== Where possible, tests will be added to the libvirt tests. Some testing ideas: * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]]: Create a round trip converting XML to native and then back, as well as native to xml and back. This isn't really practicable since we lose information. Tests have been implemented and/or updated for: * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLFromNative|virConnectDomainXMLFromNative]] * [[http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainXMLToNative|virConnectDomainXMLToNative]]: * bootrom support ===== The Code ===== ==== WIP ==== I have a [[https://github.com/fabianfreyer/libvirt/tree/gsoc/bhyveConnectDomainXMLFromNative|Fork on GitHub]] of the libvirt project, where I am pushing my code. The following branches are currently used libvirt: * [[https://github.com/fabianfreyer/libvirt/tree/gsoc/bhyve_lpc_bootrom|gsoc/bhyve_lpc_bootrom]]: implement support for booting via UEFI firmware. Posted to Mailinglist: [[https://www.redhat.com/archives/libvir-list/2016-June/msg02216.html|v1]] * [[https://github.com/fabianfreyer/libvirt/tree/gsoc/bhyve_gop|gsoc/bhyve_gop]]: implement support for VNC framebuffer * [[https://github.com/fabianfreyer/libvirt/tree/gsoc/fix/bhyveCapsBuild|gsoc/fix/bhyveCapsBuild]]: fix emulator binary in virBhyveCapsBuild virt-manager * [[https://github.com/fabianfreyer/virt-manager/tree/gsoc/bhyve-fixes|gsoc/bhyve-fixes]] Ports: * [[https://reviews.freebsd.org/D7223|D7223]]/[[https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211154|#211154]]: allow USE_GCC dependencies to be specified individually for {BUILD_,RUN_,TEST_}DEPENDS. * [[https://reviews.freebsd.org/D7225|D7225]]/[[https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211157|#211157]]: metaport for bhyve firmware I have also created a patch to bhyve_graphics: * [[https://reviews.freebsd.org/D7029|D7029]] ==== Upstreamed ==== === Connection Calls === * [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=126e630e85311a57f87709a269bc55059cb3cf44|126e630]] virConnectGetType: Trivially return ''%%"BHYVE"%%'' * [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=a1efc9428bcdf6075fac0dac4d73ff2858216d7f|a1efc94]] virConnectIsAlive: Trivially return 1, since "A connection will be classed as alive if it is [...] local" and ''%%/dev/vmm%%'' is local. * [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=32aa9ed3ba2117936b8634091330418573d9b8c8|32aa9ed]] virConnectIsEncrypted: Trivially return 0, since bhyve does not support encrypted interfaces. * [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=ef45eb9bc7782a1edb7237bb0307afe576d287d4|ef45eb9]] virConnectIsSecure: Trivially return 1, since ''%%/dev/vmm%%'' is considered a "local connection". * [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=b436a8ae5ccb04f8cf893d882d52ab5efc713307|b436a8a]] to [[https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=dd23c3822be7711fb2b25badb3a6854d9ccd1110|dd23c38]] [[https://github.com/fabianfreyer/libvirt/tree/gsoc/bhyveConnectDomainXMLFromNative|gsoc/bhyveConnectDomainXMLFromNative]] Posted to Mailinglist: [[https://www.redhat.com/archives/libvir-list/2016-June/msg00001.html|v1]] [[https://www.redhat.com/archives/libvir-list/2016-June/msg00728.html|v2]] [[https://www.redhat.com/archives/libvir-list/2016-June/msg01741.html|v3]] [[https://www.redhat.com/archives/libvir-list/2016-June/msg02138.html|v4]] [[https://www.redhat.com/archives/libvir-list/2016-July/msg00257.html|v5]] * [[https://libvirt.org/git/?p=libvirt.git;a=commit;h=32916b1699489584f66a53eaac322313803cabcc|32916b1]] [[https://github.com/fabianfreyer/libvirt/tree/gsoc/bhyveConnectGetDomainCapabilities|virConnectGetDomainCapabilities]] Posted to Mailinglist: [[https://www.redhat.com/archives/libvir-list/2016-July/msg00263.html|v1]] === Bugfixes === * [[http://libvirt.org/git/?p=libvirt.git;a=commit;h=843a70a822739cee338863f0210916361feecfc2|843a70a]] [[https://github.com/fabianfreyer/libvirt/tree/gsoc/fix/test-wrap-argv.pl|gsoc/fix/test-wrap-argv.pl]] Posted to Mailinglist: [[https://www.redhat.com/archives/libvir-list/2016-July/msg00255.html|v1]] === Ports === * [[http://www.freshports.org/sysutils/uefi-edk2-bhyve|sysutils/uefi-edk2-bhyve]] * [[http://www.freshports.org/sysutils/uefi-edk2-bhyve-csm|sysutils/uefi-edk2-bhyve-csm]] ===== Useful links ===== The following contributions are related to my work: * [[https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210111|[bhyve [patch print list of supported PCI and LPC devices]] by Roman Bogorodskiy * [[https://reviews.freebsd.org/D6999|D6999: Make kern.argmax boot tunable]] fixes make syntax-check on %%FreeBSD%%.