VMWare and other PC emulation/virtualization software (doesn't matter if they make use of “hardware supported VMM” in Intel and AMD CPUs) generally have the following properties that make them a bad choice to run if performance is benchmarked:

• Timer issues. It takes much longer to generate and process a timer (clock) event under emulation than on native hardware. Since the host OS can run with timer set at any frequency, virtualization software must first emulate generating timer ticks (interrupts), then emulate the OS's handling. All this leads to having timer interrupts generate much larger impact in the workloads measured than in 'native' case. Timers can be skewed! See this and this official VMWare documents for similar issues, and this for an error report.
• CPU cache trashing. Using virtualization software there's a lot more context switches compared with only the native kernel running. In addition to “obvious” things (like ATA interrupt first occuring on the host OS and is then passed through the host kernel, and then is emulated on the guest kernel) there are more subtle issues like having normally non-preemptable things in the guest OS preempted by the VMM.
• Privileged instruction emulation. Most privileged CPU instructions are caught by the VMM and emulated in software (and are thus slower) rather than passed to hardware. For an example of instruction emulation (that's apparently fixed in later versions of VMWare) see here.
• Non-repeatability. Interested parties cannot reliably reproduce your results if they don't have the same exact combination of VMM version and hardware. Bugs and performance issues are fixed from one version of VMWare to the other. Performance characteristics of an OS running under VMWare is different than when running on native hardware.
• Hard to be in control of the environment. In the normal case (without VMM) it's usually simpler to control the benchmark environment - cron and syslog daemons can be moved or stopped, unused drivers can be disabled, etc. Running on a VMM means all these factors need to be accounted for on both the host and the guest OS. In case the host OS is a “encumbered” OS like Windows or a recent Linux that boots into GUI, it's harder to control the background processes (on the host) that might kick in right in the middle of benchmarking (not impossible though).
• Load moderation. Hypervisors of all types can (and do) throttle CPU, disk and network IO as they see fit to balance across all of the VMs. This introduces hidden unaccountable latencies.
• Limited hardware support. Hardware emulated on VMM is not real world hardware - except for possibly the disk controllers, hardware emulated in VMWare can no longer be bought anywhere.

Of course, if you're planning to deploy FreeBSD on VMWare, it makes sense to do benchmarking on the target configuration. If not, and if using VMWare is unavoidable, please make a note saying “benchmarked under VMWare x.y” or similar in your benchmark results / mailing list posting / web page / etc.

CategoryVirtualization