====== bhyve VM usage in Jenkins cluster ======

===== Installation of FreeBSD into bhyve VM for use in Jenkins cluster =====

  - Make sure that in **/boot/loader.conf** you have:

<code>
# Include vmm driver for bhyve
vmm_load="YES"
# Load nmdm driver to communicate with bhyve VM's
nmdm_load="YES"
# For bhyve VM's networking
if_bridge_load="YES"
bridgestp_load="YES"
if_tap_load="YES"
</code>

  - Obtain a release ISO image from [[http://www.freebsd.org/where.html|FreeBSD download site]].
  - Create a ZFS dataset which will hold the VM

<code>
zfs create zroot/vm
</code>

  - Set ZFS property so that dataset will export ZVOL's as block devices only, and not try to probe them with GEOM:

<code>
zfs set volmode=dev zroot/vm
zfs get volmode zroot/vm
</code>

  - Create a 200G zvol:

<code>
zfs create -V 200G zroot/vm/disk.img
</code>

  - Boot the release ISO image and install it into the disk image:

<code>
sh /usr/share/examples/bhyve/vmrun.sh -i -I FreeBSD-9.2-RELEASE-amd64-disc1.iso -d /dev/zvol/zroot/vm/disk.img -m 2G vm_test
</code>

  - Do the initial installation, using **disk.img** as the backing hard drive
  - Edit **/etc/ttys** and add this line:

<code>
ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure
}}} or this, if the guest is running 11-CURRENT newer than r264175 {{{
ttyu0  "/usr/libexec/getty 3wire"  vt100  on  secure
</code>

  - shutdown the VM
  - Reboot the VM without the ISO image:

<code>
sh /usr/share/examples/bhyve/vmrun.sh -d disk.img -m 2G vm_test
</code>

  - Set up basic networking inside the VM by editing **/etc/rc.conf**
  - Reboot the VM.
  - Install necessary packages such as [[http://www.freshports.org/devel/jenkins/|devel/jenkins]]

===== Configure tap devices for bhyve VM's =====

  - On the host machine which is running the VM's, pick a network interface which will be used in the bridge.  This example will use **igb0**.
  - Add the following to **/etc/rc.conf** on the host machine:

<code>
#####################################################
# Create tap devices, one tap interface per bhyve VM.
# Add the tap interfaces to bridge0
####################################################
cloned_interfaces="bridge0 tap0 tap1"

autobridge_interfaces="bridge0"
autobridge_bridge0="tap* igb0"
</code>

  - Each VM should have a separate tap device, all connected to the same bridge.  If you add more VM's, remember to add another tap device to **cloned_interfaces**
  - Add the following to **/etc/sysctl.conf** on the host machine:

<code>
# bhyve needs this for tap interfaces
net.link.tap.user_open=1
net.link.tap.up_on_open=1
</code>

  - Refer to: http://www.freebsd.org/doc/handbook/network-bridging.html

===== Accessing console of VM =====

  - Type the following to see the list of VM's running:

<code>
ls /dev/vmm/*
</code>

  - The bhyve VM's are configured to send their console output to the [[http://www.freebsd.org/cgi/man.cgi?query=nmdm&sektion=4|nmdm(4)]] nullmodem driver.  Type the following to see the list of nullmodem devices in use:

<code>
ls /dev/nmdm*
</code>

  - Each [[http://www.freebsd.org/cgi/man.cgi?query=nmdm&sektion=4|nmdm(4)]] has two endpoints **A** and **B**.  You must connect to the **B** endpoint using the [[http://www.freebsd.org/cgi/man.cgi?query=cu&sektion=1|cu]] utility.  For example:

<code>
cu -l /dev/nmdm0B
</code>

  - To disconnect, type:

<code>
~..
</code>

===== Startup scripts =====

The scripts used to start the bhyve VM's are located at Github: https://github.com/freebsd/freebsd-ci/tree/master/vm

===== Configure rc.d scripts to boot VM =====

  - Read this: [[https://github.com/freebsd/freebsd-ci/tree/master/vm|README for VM scripts]]