===== Introduction =====
bhyve, pronounced "beehive" is a hypervisor/virtual machine manager for %%FreeBSD%% that supports most Intel and AMD processors that report the "POPCNT" (POPulation Count) processor feature in dmesg(8).
Most Intel Atom C2000, Celeron, Core i3, i5, i7 and related Xeon processors support bhyve but early "Core" processors may only support one virtual machine.
Most "[[http://en.wikipedia.org/wiki/AMD_10h|Barcelona]]" class and newer AMD processors support bhyve but some, notably the "Kuma" core processors include POPCNT but lack the required "NRIPS" (Next RIP Save) feature.
===== Events =====
* [[bhyve/OfficeHours|Office Hours]] Monthly Meeting
===== Presentations =====
* linux.conf.au 2020 - Peter Grehan - %%FreeBSD%% bhyve intro - [[https://www.youtube.com/watch?v=aFaLkxwvYZw|YouTube]]
* %%MeetBSD%% 2016 - Michael Dexter: Life of bhyve - [[https://www.youtube.com/watch?v=YHETy7Km7S8|YouTube]]
* BSDNow Episode 149 - Interview with Michael Dexter - [[http://www.bsdnow.tv/episodes/2016_07_06-a_wild_dexter_appears|Podcast]]
* bhyvecon Tokyo 2016 - Kris Moore: An introduction to the PC-BSD sysadm/bhyve API - [[http://bhyvecon.org/bhyvecon2016-Kris.pdf|PDF]]
* bhyvecon Tokyo 2016 - Mike Larkin: %%OpenBSD%% vmm Update - [[http://bhyvecon.org/bhyvecon2016-Mike.pdf|PDF]]
* bhyvecon Tokyo 2016 - Reyk Floeter: %%OpenBSD%% vmd Update - [[http://bhyvecon.org/bhyvecon2016-Reyk.pdf|PDF]]
* bhyvecon Tokyo 2016 - Mihai Carabas: bhyve/ARM Update - [[http://bhyvecon.org/bhyvecon2016-Mihai.pdf|PDF]]
* bhyvecon Tokyo 2016 - Roger Pau Monné: %%FreeBSD%% Xen x86/ARM Update - [[http://bhyvecon.org/bhyvecon2016-Roger.pdf|PDF]]
* %%EuroBSDcon%% 2015 - %%FreeBSD%% Virtualisation Options [[https://www.youtube.com/watch?v=Tqj71d7bw2o&feature=youtu.be&t=7146|YouTube]]
* %%EuroBSDcon%% DevSummit 2015 [[DevSummit/201510/Virtualization|Wiki Page]]
* vBSDcon 2015 - %%FreeBSD%% Virtualization Options and %%BoF%% [[https://www.youtube.com/watch?v=JgsBnLFfZaE|YouTube]]
* BSDCan 2015 - %%FreeBSD%% Virtualization Options and %%BoF%% [[https://www.bsdcan.org/2015/schedule/events/558.en.html|Schedule]]
* SCALE 2015 - %%FreeBSD%% Virtualization Options [[https://www.youtube.com/watch?v=FksmK_ORitI|YouTube]] [[https://www.youtube.com/watch?v=gsoFObZNS1k|Video 2]]
* bhyvecon Tokyo 2015 - What's new in bhyve by Peter Grehan - [[http://bhyvecon.org/bhyvecon2015-Peter.pdf|PDF]]
* bhyvecon Tokyo 2015 - An Introduction to bhyveucl by Allan Jude
* bhyvecon Tokyo 2015 - Xen virtualization on %%FreeBSD%% by Roger Pau Monné - [[http://bhyvecon.org/bhyvecon2015-Roger.pdf|PDF]]
* bhyvecon Tokyo 2014 - Introduction to bhyve - [[https://bhyvecon.org/introduction_to_bhyve.pdf|PDF]] - [[http://www.youtube.com/watch?v=MTXUZLpCvfg|YouTube]]
* bhyvecon Tokyo 2014 - bhyve past, present, future - [[http://bhyve.org/presentations/bhyve-past_present_future.pdf|PDF]] - [[http://www.youtube.com/watch?v=lTOiSyu0-MA|YouTube]]
* bhyvecon Tokyo 2014 - bhyve Provisioning and Monitoring - [[http://bhyve.org/presentations/bhyve_provisioning_and_monitoring.pdf|PDF]] - [[http://www.youtube.com/watch?v=aDEzisLndE4|YouTube]]
* bhyvecon Tokyo 2014 - Depenguinating Your Infrastructure - [[http://bhyve.org/presentations/depenguinating_your_infrastructure.pdf|PDF]] - [[http://www.youtube.com/watch?v=yNxQQTFphpM|YouTube]]
* bhyvecon Tokyo 2014 - OSv on bhyve - [[http://bhyve.org/presentations/osv_on_bhyve.pdf|PDF]] - [[http://www.youtube.com/watch?v=nUDApXhwtCc|YouTube]]
* bhyvecon Tokyo 2014 - ruby-virtualmachine - [[http://bhyve.org/presentations/ruby-virtualmachine.pdf|PDF]] - [[http://www.youtube.com/watch?v=Foeou4T4ZR4|YouTube]]
* bhyvecon Tokyo 2014 - Introduction to Qubes OS - [[http://bhyve.org/presentations/introduction_to_qubes_os.pdf|PDF]] - [[http://www.youtube.com/watch?v=TMDqAfvDh3Q|YouTube]]
* bhyvecon Tokyo 2014 - Mocloudos - [[http://bhyve.org/presentations/mocloudos.pdf|PDF]] - [[http://www.youtube.com/watch?v=01-OKdSb-sg|YouTube]]
* BAFUG March 2014 - [[http://www.ixsystems.com/whats-new/jenkins-and-bhyve-continuous-integration-for-freebsd-by-craig-rodrigues/|Jenkins and bhyve]]
* %%AsiaBSDCon%% 2014 - [[http://2014.asiabsdcon.org/timetable.html.en#P6A|Nested Paging in bhyve]]
* FOSDEM 2014 - What's new in %%FreeBSD%% 10? [[http://www.youtube.com/watch?v=2hU6tLw6GFA|YouTube]]
* BSDNow Episode 020 - bhyve Mind [[http://www.bsdnow.tv/episodes/2014_01_15-bhyve_mind|Podcast]]
* %%CapBUG%% - [[http://www.capbug.org/team-announcements/novemberdecember2013|Live Demo]]
* Sysadmin #5 in Paris - [[http://people.freebsd.org/~bapt/sysadmin.pdf|PDF]]
* November 2013 %%FreeBSD%% DevSummit - bhyve Update - [[http://people.freebsd.org/~grehan/talks/vendor-summit.pdf|PDF]]
* %%OpenFest%% 2013 - bhyve - the BSD hypervisor [[http://openfest.org/schedule/]] - [[http://bhyve.org/presentations/openfest2013-handout.pdf|PDF]] - [[https://www.youtube.com/watch?v=XFacfnnPB9c|YouTube]]
* %%EuroBSDcon%% 2013 - %%FreeBSD%% bhyve Hypervisor hosting Other Systems [[http://2013.eurobsdcon.org/eurobsdcon-2013/talks/#PeterGrehan|Schedule]] - [[http://people.freebsd.org/~grehan/talks/eurobsdcon_2013_bhyve.pdf|PDF]]
* %%TechSNAP%% 111 - BSDCan Recap [[http://www.jupiterbroadcasting.com/37661/bsdcan-2013-recap-techsnap-111/|Podcast]]
* BSDCan 2013 - Hands-On bhyve, the BSD Hypervisor [[http://www.bsdcan.org/2013/schedule/events/375.en.html|Schedule]] - [[http://bhyve.org/presentations/bhyve-BSDCan2013.pdf|PDF]]
* %%AsiaBSDCon%% 2013 - Hands-On bhyve, the BSD Hypervisor [[http://2013.asiabsdcon.org/timetable.html]] - [[https://2013.asiabsdcon.org/papers/abc2013-P4A-paper.pdf|PDF]]
* %%AsiaBSDCon%% 2013 - Implements BIOS emulation support for %%BHyVe%%: A BSD Hypervisor [[http://2013.asiabsdcon.org/timetable.html|Schedule]]
* bhyve on BSDTalk - [[http://bsdtalk.blogspot.com/2013/02/bsdtalk223-bhyve-with-michael-dexter.html|MP3 and OGG]]
* FOSDEM 2013 - [[https://fosdem.org/2013/schedule/event/bhyve_bsd_hypervisor/|BSD devroom talk]] - [[https://archive.fosdem.org/2013/schedule/event/bhyve_bsd_hypervisor/attachments/slides/279/export/events/attachments/bhyve_bsd_hypervisor/slides/279/bhyve_slides.pdf|PDF]]
* %%EuroBSDcon%% 2012 - [[http://bhyve.org/BHyVe-EuroBSDcon2012.pdf|PDF]] - [[http://www.youtube.com/watch?v=4d0rLrgArHw|YouTube]]
* %%MeetBSD%% California 2012 - [[http://bhyve.org/BHyVe-MeetBSD2012.pdf|PDF]]
* BSDCan 2012 - BSD Multiplicity [[http://www.bsdcan.org/2012/schedule/events/291.en.html|Schedule]]
* BSDCan 2011 - %%BHyVe%% BSD Hypervisor {{devsummit/201105/bhyve.pdf?linkonly|PDF}} - [[http://www.bsdcan.org/2011/audio/BHyVeNativeBSDHypervisor.mp3|MP3]]
===== Frequently Asked Questions =====
==== General ====
=== Q: Is it "bhyve", "Bhyve", "BHyVe" or BHyve? ===
A: The developers mercifully retired "%%BHyVe%%" and simply refer to it as "bhyve", after the utility.
=== Q: How do I configure bhyve and install a VM operating system? ===
A: Begin with the %%FreeBSD%% Handbook [[http://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html|chapter]] and [[https://man.freebsd.org/cgi/man.cgi?query=bhyve|manual page]].
=== Q: Could bhyve be ported to other operating systems? ===
Yes. See the [[https://github.com/mist64/xhyve|xhyve]], [[http://www.pluribusnetworks.com/products/netvisor-os/|Pluribus Netvisor]], and [[http://bhyvecon.org/bhyvecon2018-Gwydir.pdf|bhyve in Illumos-based distributions]] efforts.
=== Q: What VM operating systems does bhyve support? ===
A: bhyve supports any version of %%FreeBSD%% i386/amd64 with %%VirtIO%% support. %%OpenBSD%%, %%NetBSD%% and GNU/Linux are supported using the ''%%sysutils/grub2-bhyve%%'' port.
[[bhyve/UEFI|UEFI support]] is available for %%FreeBSD%%, %%OpenBSD%%, GNU/Linux, %%SmartOS%% and Windows:
* %%FreeBSD%% 8.4-RELEASE and newer, including CURRENT
* [[bhyve/OpenBSD|OpenBSD amd64/i386 5.2 and newer]]
* GNU/Linux amd64/i386 (Many distributions)
* %%NetBSD%% amd64 6.1 and newer
* %%SmartOS%% 20151001 and newer
* [[bhyve/Windows|Windows x64 Vista, 7, 8, 10, 11]]
* Windows Server 2008r2, 2012r2, 2016
* [[bhyve/9front|9front]]
* [[bhyve/OmniOS|OmniOS]]
Tested GNU/Linux distributions include:
* [[bhyve/Gentoo|Gentoo]]
* %%CentOS%%/RHEL 6.2, 6.3, 6.4, 6.5 and 7.0
* Debian 6.0.7, 7.0, 7.6 netinstall i386/amd64
* Fedora 20
* openSUSE 12.3 and 13.1 amd64
* Ubuntu Server 10.04, 12.04, 13.04, 14.04, 14.10, 15.04, 15.10, 16.04 i386/amd64
* Void Linux
See the [[https://github.com/grehan-freebsd/grub2-bhyve|grub2-bhyve]] page for detailed ''%%sysutils/grub2-bhyve%%'' instructions and for best results use %%FreeBSD%% CURRENT with the -w bhyve flag.
=== Q: What does bhyve "look" like? ===
To load a %%FreeBSD%% kernel from disk image ''%%vm0.img%%'' with 256M RAM and the name ''%%vm0%%'':
/usr/sbin/bhyveload -m 256 -d ./vm0.img vm0
This will show the %%FreeBSD%% loader screen and you should see the device ''%%/dev/vmm/vm0%%''
To boot the VM with 2 vCPUs, the same 256M RAM and the ''%%tap0%%'' network interface:
/usr/sbin/bhyve -c 2 -m 256 -A -H -P \
-s 0:0,hostbridge \
-s 1:0,virtio-net,tap0 \
-s 2:0,ahci-hd,./vm0.img \
-s 31,lpc -l com1,stdio \
vm0
After the VM has been ''%%shutdown%%'', its resources can be reclaimed with:
/usr/sbin/bhyvectl --destroy --vm=vm0
=== Q: What is the easiest way to try bhyve? ===
The included method booting %%FreeBSD%% virtual machines is the ''%%/usr/share/examples/bhyve/vmrun.sh%%'' script:
fetch ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/11.0-STABLE/amd64/Latest/FreeBSD-11.0-STABLE-amd64.raw.xz
unxz FreeBSD-11.0-STABLE-amd64.raw.xz
sudo sh /usr/share/examples/bhyve/vmrun.sh -d FreeBSD-11.0-STABLE-amd64.raw vm0
%%FreeNAS%% also offers a graphical bhyve management interface: http://www.freenas.org/
=== Q: What is required to support Graphical UEFI and operating systems like Microsoft Windows? ===
See the [[bhyve/UEFI]] and [[bhyve/Windows|bhyve Windows Virtual Machines]] wiki pages.
=== Q: Does libvirt support bhyve? ===
A: [[http://libvirt.org/drvbhyve.html|Yes]]
=== Q: Are there other frameworks for bhyve? ===
A: There is [[http://vmrc.bsd.lv|vmrc]] among others, a search of [[https://github.com/search?q=bhyve|Github]] will reveal several.
=== Q: Can I run multiple bhyve hosts under VMware nested VT-x EPT? ===
A: Yes, but you must [[https://communities.vmware.com/docs/DOC-8970|enable it]]: look for the GUI or ''%%vhv.allow%%'' or ''%%vhv.enable%%'' options, depending on your version of VMware. VMware Fusion: Processors & Memory: Advanced options: Enable hypervisor applications in this virtual machine
Confirmed with VMware Workstation, VMware Player, VMware Fusion and ESXi. Note that you must stop all running VMs before you can enable EPT pass-through.
=== Q: Can I run multiple bhyve hosts under Linux KVM with nested VT-x EPT? ===
A: Maybe. You must [[http://www.rdoxenham.com/?p=275|enable]] it and note the [[https://www.kernel.org/doc/Documentation/virtual/kvm/nested-vmx.txt|known limitations]]. Please let us know if you are successful.
=== Q: Can I run bhyve and VirtualBox on FreeBSD at the same time? ===
A: No. See [[http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-August/001438.html|this thread]] for more information.
=== Q: Do VirtualBox or Hyper-V support nested VT-x EPT? ===
A: VirtualBox [[https://www.virtualbox.org/ticket/4032|does not]], but Hyper-V has nested EPT available to [[https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting|Windows Insiders]].
=== Q: Does Debian GNU/kFreeBSD support bhyve? ===
A: [[https://wiki.debian.org/Debian_GNU/kFreeBSD|Good question!]]
=== Q: Can I use disk images from VirtualBox or other virtualization platforms on bhyve? ===
A: bhyve does not currently support reading foreign disk image formats directly, so one will have to convert existing disk images to a raw image using qemu-img. This can be accomplished by doing the following on a %%FreeBSD%% system:
$ pkg install qemu-devel
$ qemu-img convert -f vmdk -O raw virtual_box_image.vmdk bhyve_raw_image.img
VMDK, QCOW, QCOW2, VDI and more images can be converted to raw images using the [[https://man.freebsd.org/cgi/man.cgi?qemu-img|qemu-img(1)]] utility in the [[https://www.freshports.org/emulators/qemu-devel/|emulators/qemu-devel]] port.
=== Q: What about OmniOS support? ===
A: See [[bhyve/OmniOS]].
==== Technical Questions ====
=== Q: What comprises bhyve? ===
A: The ''%%vmm.ko%%'' loadable kernel module, the ''%%libvmmapi%%'' library, the ''%%bhyve%%'', ''%%bhyveload%%'' and ''%%bhyvectl%%'' utilities.
In total these binaries are under 500K in size.
=== Q: Does bhyve support VT-d PCI device pass-through? ===
A: Yes, on Intel CPUs. See [[bhyve/pci_passthru]].
=== Q: Does bhyve support AHCI devices? ===
A: Yes, and they provide non-blocking I/O for higher performance.
CD: ''%%-s 4,ahci-cd,/path/to/image.iso%%''
HD: ''%%-s 4,ahci-hd,/path/to/disk.img%%''
=== Q: Does bhyve have an "out-of-band"/"lights-out management" serial console? ===
A: Yes, and it can be accessed several ways including as a ''%%nmdm%%'' nullmodem terminal or a pty. ''%%stdio%%'' output can also be sent to a terminal multiplexer like ''%%tmux%%'' or ''%%screen%%''. Example ''%%nmdm%%'' syntax is:
''%%-s 31,lpc -l com1,stdio%%''
Becomes:
''%%-s 31,lpc -l com1,nmdm0A%%''
Before starting the VM, run:
''%%kldload nmdm%%''
Start the VM but note that it will not display any output when booted. Connect to the serial interface with:
''%%cu -l /dev/nmdm0B -s 9600%%''
Notes:
A is connected to B
''%%nmdmN%%'' is a a 32 bit unsigned integer so you have plenty to use.
''%%ptys%%'' can be created with ''%%socat%%''.
This functionality is included in the %%FreeBSD%% 10.0 release.
=== Q: Is there any way to determine if an OS is virtualized? ===
A: Yes, the ''%%kern.vm_guest%%'' sysctl will report if a %%FreeBSD%% guest is under KVM or bhyve.
=== Q: Where are the sources? ===
A: The source is in %%FreeBSD%% SVN source repository: sys/amd64/vmm/, usr.sbin/bhyve/, usr.sbin/bhyveload/, usr.sbin/bhyvectl/, and lib/libvmmapi/
==== Troubleshooting ====
=== Q: What will I see if my CPU does not support EPT? ===
# kldload vmm
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6
=== Q: What will I see if EPT is disabled in BIOS? ===
# sh /usr/share/examples/bhyve/vmrun.sh ...
Launching virtual machine "...
vm_create: Device not configured
=== Q: What will I see if a VM name is over 31 characters long? ===
vmcreate: Invalid argument
=== Q: Why does my system appear to stop booting just after "Starting background file system checks in 60 seconds."? ===
A: This indicates that your ''%%/etc/ttys%%'' is not properly configured for serial output for use with bhyve.
=== Q: What is the string I need to add to "/etc/ttys" for older versions of FreeBSD?" ===
ttyu0 "/usr/libexec/getty 3wire.9600" vt100 on secure
=== Q: Why won't vmrun.sh VMs archived with tar? ===
A: ''%%vmrun.sh%%'' creates disk images using the ''%%truncate(1)%%'' command and as per the ''%%tar(1)%%'' manual page, "There is not yet any support for multi-volume archives or for archiving sparse files."
=== Q: Why can't I network VMs over a wireless host interface? ===
A: Access points cannot determine the destination VMs and only see the host.
=== Q: Why does mounting my FreeBSD with ZFS on root VM fail with "error 45" when "Mounting from zfs:" ? ===
A: You are probably attempting to boot an older version of %%FreeBSD%% on a zpool that you created with a newer version of %%FreeBSD%% and the available ZFS features are not available. A safe set of features to use with %%FreeBSD%% 9.2 and 10.0 are:
zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ...
=== Q: Why does my VM fail with "vm_run error -1, errno 25" ? ===
A: This can happen if your kernel and world are out of sync.
=== Q: Are there any debugging options for bhyve? ===
A: Yes: The [[bhyve/DebuggingWithGdb|gdb]] interface.
=== Q: Why can't my nested VMs reach the network? ===
A: The bottom most hypervisor needs to run the virtual interface in promiscuous mode in order for packets destined for an ethernet address other than that of the first level guest to be received by the virtual interface.
[[http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004099|Enabling Promiscuous mode in VMWare ESX]]
==== Participation ====
=== Q: Is there a mailing list or IRC channel? ===
A: The most popular places to discuss bhyve are the ''%%freebsd-virtualization@freebsd.org%%'' [[http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization|mailing list]] and the ''%%#bhyve%%'' channel on irc.libera.chat.
=== Q: How can I help? ===
A: First and foremost, by testing bhyve on your unique hardware and in every configuration you can imagine. There could easily be remaining bugs and it is in everyone's best interests for these to be resolved ASAP.
If you think you have an issue, considering asking about it on IRC. If you know you have an issue, consider posting it on the %%FreeBSD%% Virtualization mailing list.
Testing ideas:
* High vCPU numbers
* High RAM allocations
* High CPU, network and storage loads
* Every piece of compatible hardware you can find
* PCI pass-through with every device you can find
* Every storage strategy you can think of including ZVOLs and iSCSI targets
* Watch for clock drift
* Test every possible VM OS
* Countless things only you will think of!
=== Q: How do I report a panic? ===
A: If your host or VM panics and enters the debugger, the information you provide can be very useful for revealing and reproducing bugs. Typing "bt" at the debugger will produce a backtrace and "show thread" on various hex strings may yield more. Please capture this information and use your best judgement in sending it to the bhyve developers or the freebsd-virtualization mailing list. If in doubt, I am happy to relay reports (address below).
Do consider using the [[https://www.freshports.org/sysutils/panicmail|sysutils/panicmail]] port with bhyve.
----
{{tag>HowTo Project Virtualization}}