====== bhyve PCI Passthrough ======

bhyve supports passing of host PCI devices to a virtual machine for its exclusive use of them.

**Note:** VGA / GPU pass-through devices are not currently supported.

====== Requirements ======

  * CPU supports Intel IOMMU (a.k.a. VT-d) feature
  * PCI device (and driver) supports MSI/MSI-x interrupts

Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with ''%%acpidump -t | grep DMAR%%''

PCI card MSI/MSI-x support can be determined with ''%%pciconf -lc | grep MSI%%''

====== Instructions ======

1. Configure vmm.ko to be preloaded at boot-time in loader.conf, adding:

<code>
vmm_load="YES"
</code>

2. Determine the bus/slot/function of the device that you want to pass through to the guest with ''%%pciconf -v -l%%''

For example, to pass through a Atheros Gigabit Ethernet adapter:

<code>
alc0@pci0:2:0:0:	class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet
</code>

The bus/slot/function is for this devices is 2/0/0 (from the end of 'alc0@pci0:2:0:0')

3. Once the desired PCI device is identified, it and any others must be masked from the host early in the boot process with a PCI passthrough devices entries or "pptdevs" in loader.conf

The PCI device at bus/slot/function 2/0/0 would be masked from the host with:

<code>
pptdevs="2/0/0"
</code>

Multiple devices can be added, space-separated, to the "pptdevs" entry

<code>
pptdevs="2/0/0 1/2/6 4/9/0"
</code>

The "pptdevs" entry supports up to 128 characters in the string. Any additional entries must be added with additional, numbered "pptdevsN" entries:

<code>
pptdevs2="123/2/0 123/3/0"
</code>

4. Rebooting the system will cause the requested PCI passthrough devices to be masked from the system and become available for use in a virtual machine

For example, to the Atheros Gigabit Ethernet adapter will be associated with device **ppt0** in the virtual machine:

<code>
ppt0@pci0:2:0:0:	class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
    vendor     = 'Atheros Communications'
    device     = 'AR8151 v2.0 Gigabit Ethernet'
    class      = network
    subclass   = ethernet
</code>

5. The slot option to the bhyve(8) command is used to bind the masked host PCI device to the guest using the b/s/f as the identifier. The guest slot/function has no correlation to the host b/s/f. For example, to assign the Atheros Gigabit Ethernet adapter to a guest at slot 7, the bhyve(8) command line would contain:

<code>
-s 7,passthru,2/0/0
</code>

Caveats: multi-function devices are not always independent, and may have to be assigned to guests with the functions being the same. An example is QLogic FC adapters, where function 0 is used for firmware loading, while the other functions are for port data transfer. For these, the mappings of functions must be the same in the guest e.g.

<code>
-s 7:0,passthru,4/0/0 -s 7:1,passthru,4/0/1 -s 7:2,passthru,4/0/2
</code>

Intel network adapters *do not* have this issue and can be split out to different guest slots e.g. an Intel adapter at host 6/5/0 and with 2 functions could be setup in the guest as two separate devices at slot 3 and slot 8:

<code>
-s 3:0,passthru,6/5/0   -s 8:0,passthru,6/5/1
</code>

====== Additional Notes ======

  * The "-S" option must be passed to both bhyveload(8) and bhyve(8) to wire guest memory.
  * bhyve support AMD-Vi/IOMMU by change [[https://svnweb.freebsd.org/base?view=revision&revision=317600|r317600]]. By default AMD-Vi passthrough support is disabled, set ''%%hw.vmm.amdvi.enable%%'' and reload vmm.ko to enable it.

----

[[/CategoryVirtualization]]
[[/CategoryHowTo]]