Xen Hypervisor Dom0 and DomU Support

This is an old revision of the document!

Xen is a GPLv2-license hypervisor for the Intel and ARM architectures

FreeBSD has included i386 and amd64 DomU and Amazon EC2 unprivileged domain (virtual machine) support since FreeBSD 8.0 and includes Dom0 control domain (host) support in FreeBSD 11.0. Support for para-virtualized (PV) domains has been removed from FreeBSD 11 in favor of hardware virtualized (HVM) domains.

FreeBSD Dom0 Control Domain Support

FreeBSD 11 includes experimental Dom0 Control Domain support as of FreeBSD 11 SVN revision r282269 and xen-tools-4.5.0_4 (r385157).

Hardware virtualized domains require Extended Page Table (EPT) and Input/Output Memory Management Unit (IOMMU) processor features to operate.

Suppressed FreeBSD kernel messages during boot. A block cursor will be visible during the FreeBSD kernel boot, eventually giving the FreeBSD login prompt
Some Intel SandyBridge systems have demonstrated Dom0 panics with X11
The kernel debugger may not work
Inconsistent SC console support
MSI-X PV interrupts are not supported (FreeBSD)
PCI devices with virtual functions work inconsistently (FreeBSD)
DomU Save, Restore and Migration is not supported (Xen)
PCI Passthrough is not supported (Xen/FreeBSD)
No Xen ARM support
Loading vmm.ko will panic the Dom0, as may VirtualBox
Potentially unstable behavior, warranting your testing

Dom0 Control Domain Installation

The 'xen' metapackage including the 'xen-kernel' and 'xen-tools' packages is supported by FreeBSD 11 amd64 binary snapshots and equivalent systems built from source. This example will assume VNC output for unprivileged domains which will be accessed from a another system using a tool such as net/tightvnc.

Xen Dom0 Package Installation

pkg install xen

Follow the instructions provided at the end of the package which modifies:

/etc/sysctl.conf
/etc/ttys
/boot/loader.conf
/boot/menu.rc.local

Reboot the host. Pressing “7” in the loader will disable booting to the Xen kernel.

You should see the Xen kernel output followed by either the FreeBSD kernel output or a blank screen until the login prompt. The FreeBSD control domain system will behave in the established manner and note the dom0_mem and dom0_max_vcpus options in /boot/loader.conf.

Unprivileged Domain Configuration

Unprivileged Domains consist of a configuration file and logical or physical optical and hard disks. For this example we will assume a ISO optical disk file freebsd.iso and a hard disk file created with truncate -s 20G freebsd.img.

An example FreeBSD DomU configuration file freebsd.cfg with 2048M RAM and two virtual CPUs without networking:

builder = "hvm"
memory = 2048
vcpus = 2
name = "FreeBSD"
disk = [ '/root/freebsd.img,raw,hda,w',
         '/root/freebsd.iso,raw,hdc:cdrom,r' ]
# boot = "c" # Boot to hard disk image
boot = "d" # Boot to ISO image
usbdevice = 'tablet'
vnc = 1
vnclisten = 0.0.0.0

To boot this DomU with -vvvv level of verbose logging:

# xl -vvvv create freebsd.cfg

# xl list
Name                        ID   Mem VCPUs      State   Time(s)
Domain-0                     0  2047     4     r-----     660.6
FreeBSD                      1  2048     2     r-----       3.5

To connect to this DomU using net/tightvnc:

# vncviewer <IP Address of the Xen Dom0>

The DomU should reboot and shutdown properly but to manually destroy it:

# xl destroy FreeBSD

An example Windows DomU configuration file windows.cfg with 4096M RAM and a two virtual CPUs without networking:

builder = "hvm"
memory = 4096
vcpus = 2
name = "Windows"
disk = [ '/root/freebsd.img,raw,hda,w',
         '/root/windows.iso,raw,hdc:cdrom,r' ]
# boot = "c" # Boot to hard disk image
boot = "d" # Boot to ISO image
vnc = 1
vnclisten = 0.0.0.0
usbdevice = 'tablet'
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
acpi = 1

Network Configuration

Xen DomU networking is fundamentally similar to bhyve VirtIO networking and to add a vif virtual network interface attached to bridge0, add to the DomU configuration file:

vif = [ 'bridge=bridge0' ]

To prepare the bridge on the Dom0 with network interface em0:

# ifconfig bridge0 create
# ifconfig bridge0 addm em0 SYNCDHCP
# ifconfig bridge0 up

This will result in the xn0 network interface appearing in the DomU.

Note that the tap and bridge devices must be present or the DomU will not boot. Add these to /etc/rc.conf to suit your unique network.

Serial Configuration

The additional configuration option serial='pty' should allow for attachment to the DomU serial console with xl create -c <DomU>.cfg. Be sure that your DomU is configured for serial console output. For a FreeBSD DomU add console="comconsole" to /boot/loader.conf. CTRL-] will disconnect from the DomU console.

Error creating domain 0 on boot indicates that VT-x extensions are unavailable or disabled in BIOS.

If your DomU “FreeBSD” does not boot, check the log in /var/log/xen/qemu-dm-FreeBSD.log

ZFS zvol-backed DomU virtual machines are supported with (must verify):

disk = ['/dev/zvol/zroot/freebsd,raw,hda,rw']
disk = ['phy:/dev/zvol/zroot/linux,xvda,w']

Some DomU virtual machines such as OpenBSD require one virtual CPU.

If your VNC display freezes or does not scale during the various DomU boot stages, try reconnecting.

DomU configuration files are not re-read on reboot.

bhyve virtual machines should boot as DomU unprivileged domains with the caveat that their network devices may change name.

FreeBSD Xen Dom0 support is an experimental feature. The freebsd-virtualization mailing list and official Xen project resources may be helpful but at this stage you may be the first person to try your given configuration.

The established Xen documentation is generally applicable to FreeBSD.