bhyve VM usage in Jenkins cluster

1. Make sure that in /boot/loader.conf you have:

# Include vmm driver for bhyve
vmm_load="YES"
# Load nmdm driver to communicate with bhyve VM's
nmdm_load="YES"
# For bhyve VM's networking
if_bridge_load="YES"
bridgestp_load="YES"
if_tap_load="YES"

1. Obtain a release ISO image from FreeBSD download site.
2. Create a ZFS dataset which will hold the VM

zfs create zroot/vm

1. Set ZFS property so that dataset will export ZVOL's as block devices only, and not try to probe them with GEOM:

zfs set volmode=dev zroot/vm
zfs get volmode zroot/vm

1. Create a 200G zvol:

zfs create -V 200G zroot/vm/disk.img

1. Boot the release ISO image and install it into the disk image:

sh /usr/share/examples/bhyve/vmrun.sh -i -I FreeBSD-9.2-RELEASE-amd64-disc1.iso -d /dev/zvol/zroot/vm/disk.img -m 2G vm_test

1. Do the initial installation, using disk.img as the backing hard drive
2. Edit /etc/ttys and add this line:

ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure
}}} or this, if the guest is running 11-CURRENT newer than r264175 {{{
ttyu0  "/usr/libexec/getty 3wire"  vt100  on  secure

1. shutdown the VM
2. Reboot the VM without the ISO image:

sh /usr/share/examples/bhyve/vmrun.sh -d disk.img -m 2G vm_test

1. Set up basic networking inside the VM by editing /etc/rc.conf
2. Reboot the VM.
3. Install necessary packages such as devel/jenkins

1. On the host machine which is running the VM's, pick a network interface which will be used in the bridge. This example will use igb0.
2. Add the following to /etc/rc.conf on the host machine:

#####################################################
# Create tap devices, one tap interface per bhyve VM.
# Add the tap interfaces to bridge0
####################################################
cloned_interfaces="bridge0 tap0 tap1"

autobridge_interfaces="bridge0"
autobridge_bridge0="tap* igb0"

1. Each VM should have a separate tap device, all connected to the same bridge. If you add more VM's, remember to add another tap device to cloned_interfaces
2. Add the following to /etc/sysctl.conf on the host machine:

# bhyve needs this for tap interfaces
net.link.tap.user_open=1
net.link.tap.up_on_open=1

1. Refer to: http://www.freebsd.org/doc/handbook/network-bridging.html

1. Type the following to see the list of VM's running:

ls /dev/vmm/*

1. The bhyve VM's are configured to send their console output to the nmdm(4) nullmodem driver. Type the following to see the list of nullmodem devices in use:

ls /dev/nmdm*

1. Each nmdm(4) has two endpoints A and B. You must connect to the B endpoint using the cu utility. For example:

cu -l /dev/nmdm0B

1. To disconnect, type:

~..

The scripts used to start the bhyve VM's are located at Github: https://github.com/freebsd/freebsd-ci/tree/master/vm

1. Read this: README for VM scripts