2015 VM Benchmarks: Parallels 11 vs. Fusion 8 vs. VirtualBox 5

Battery Life

If you read our initial looks at Parallels 11 and Fusion 8, you know that battery life was one key factor we didn’t include in our results. That’s not because we don’t think it’s important; it’s simply that it takes so darn long to run our tests, with three full tests performed for each platform and each usage scenario (that’s 24 complete drain and recharge cycles, for those who are counting). We also hit an unusual issue, described later on, that required another few rounds of testing and pushed this final article out a few days. But never fear! We’re back with battery life benchmarks for Parallels, Fusion, and VirtualBox, and we also have native performance in Boot Camp as a baseline for comparison.

With the ever increasing use of laptops and mobile computing devices, battery life is one of the most important areas of product development and the customer experience. Although energy efficiency and battery capacity have improved dramatically in recent years, it still takes a lot of power to run a complex modern operating system, and with any of the virtualization platforms we’re looking at today, we’re asking our MacBook to go even further and run two operating systems (at least!).

We obviously expect that our MacBook will experience shorter running times when using a virtual machine in OS X, but the question is how much shorter. To test that question, we turned to Futuremark Powermark, a battery life benchmark for Windows that’s based on tests similar to those conducted by its sister app, PCMark. Unlike PCMark, however, which runs a particular scenario just three times in order to determine a system’s performance level, Powermark runs the designated usage scenario repeatedly until the battery is drained.

Powermark has its own internal battery life calculator that reports both actual running time and estimated battery life remaining, but we didn’t want to take the chance that the virtualization process could introduce inaccuracies in that counter. We therefore turned to our old standby for measuring battery life in OS X: Automator. Using a basic Automator workflow, we told the script to simply write a timestamp to a text file on the OS X desktop once every 30 seconds. We then ran our Powermark tests until the battery was drained, plugged the Mac back in, booted up, and calculated the total running time based on the difference between the first and last timestamp in the file generated by Automator. With this method, which has virtually no impact on battery life, we get a reliable result of total running time that has, at most, a 30-second margin of error.

We ran two Powermark benchmarks: Productivity and Entertainment. The productivity benchmark is likely to be the most common usage scenario for a real user’s virtual machine, and tests activities like Web browsing and office apps. It should be relatively light on our Mac’s battery and offer usable running time when out and about. The Entertainment test, on the other hand, is much harder on the battery, with tasks like video playback and entry-level 3D gaming. The results from this test should give you a good idea of how long your MacBook’s battery will last while running more demanding apps in your virtual machines.

The results reported for each test, below, are measured in minutes, so longer bars are better. All tests were performed with the MacBook’s screen set to 50 percent brightness and no other apps open except for the Powermark suite in the Windows VM and our Automator script in OS X.

It should first be noted that Windows running in Boot Camp isn’t as energy efficient as OS X natively, due to the fact that Apple, which makes its own hardware, can fine-tune and optimize every aspect of hardware and software to maximize battery life. For example, we achieved an average time of 6 hours and 43 minutes while running the lightweight productivity test in Boot Camp. That’s about 16 percent less than the roughly 8 hours that we could expect from running a similarly demanding test natively in OS X. Therefore, the Boot Camp results here aren’t meant to represent the maximum battery life potential of the MacBook Pro overall, but rather serve only as a baseline for comparing native Windows performance to that of the virtual machines on the same hardware.

That said, Parallels and Fusion again trade places depending on the workload. All VMs fall notably short of the battery life attained by Boot Camp, but Parallels 11 hits an average of 5 hours and 49 minutes on the Productivity test, which is 13 percent shorter than Boot Camp and about 5 percent longer than Fusion 8.

The roles reverse for the more demanding Entertainment test. Here, even Boot Camp only hits an average of 3 hours and 25 minutes, while Fusion 8 takes second place with an average running time of 2 hours and 46 minutes, about 12 percent longer than Parallels 11.

As for VirtualBox, you shouldn’t expect to get much more than a partial afternoon’s work done, even in lightweight “productivity” scenarios, with the VirtualBox VM only hitting 3 hours and 7 minutes at best. When you add more complexity from the Entertainment test, you’re down to just 1 hour and 44 minutes.

Parallels 11 Travel Mode

One of the new features in Parallels 11 is “Travel Mode,” an option that takes over Windows’ power management settings and can automatically pause or disable resource intensive services and processes in order to reduce energy usage and extend battery life. When we set out to conduct the tests in the first chart above, we naturally enabled travel mode for our Parallels VM, as we generally want to run our tests under conditions that take advantage of each app’s unique features and settings. But when the initial results came in, we found the battery life achieved by Parallels to be surprisingly, even suspiciously, low.

As part of our troubleshooting process to figure out exactly what was going on, we eventually tried a test without Travel Mode and, a few hours later, received a more expected result like those recorded for Parallels in the first chart. Some additional testing led us to the conclusion that Parallels’ Travel Mode really shouldn’t be enabled if you’re actively trying to use your virtual machine. It seems that our battery life benchmark tests, like any running application, were constantly calling on Windows to deliver power and performance, and Parallels Travel Mode was constantly fighting that request, causing an inefficient ramping up and down of resources that actually drained the battery much quicker and resulted in slow performance from our benchmark apps (e.g., word processor windows were slow to resize, the gaming part of the benchmark ran at single-digit frame rates).

So what’s Travel Mode for, then? Well, as we continued to experiment with this issue over the past few days, we discovered that Travel Mode really can come in handy, but only when your Windows VM is in an idle state. We accidentally discovered this when one of our benchmark runs crashed at the start of the benchmark, but after we had left the room, resulting in a “test” that was nothing more than Automator writing a timestamp every 30 seconds. We returned the next morning to find a timestamp-calculated result indicating significantly better performance than any other test. After a few more runs to verify this, we concluded that the key to Travel Mode is not to fight it, and that you shouldn’t use it at all if you’re running anything in Windows other than possibly a simple text editor.

Following that advice gives us the much more interesting results, with all systems tested in the same “idle” mode:

That’s right, when your VM and Mac are both in an idle state, Parallels 11 in Travel Mode gets even better battery life than native Boot Camp (likely owing to the aforementioned energy efficiency advantages of OS X over Windows, and remembering that while in Travel Mode, the Parallels VM is just like any other energy efficient OS X app). The bad news? Your VM and Mac are in an idle state.

It’s true that you want your virtual machine to be as energy efficient as possible, especially when it’s not in use, and that may have been Parallels’ motivation for introducing this feature. But if your VM is going to be idle long enough for Travel Mode to make a difference (and, remember, it has to be completely idle, without even moderately-intensive applications running), you might as well just suspend the VM, something that Parallels is pretty good at, to preserve battery life even further. In short, Travel Mode works great in this idle testing scenario (that’s a 20 percent improvement over Parallels 11 without Travel Mode enabled), but we honestly have no idea who will find this feature useful in its current state.

Table of Contents

[one_half padding=”0 5px 20px 0″]
1. Introduction
2. Test Setup & Methodology
3. Geekbench
4. 3DMark
5. FurMark OpenGL
6. Cinebench R15
7. PCMark 8
8. Passmark PerformanceTest
[/one_half]

[one_half_last padding=”0 0px 20px 5px”]
9. Video Encoding
10. File Transfers
11. USB 3.0 Speed
12. Virtual Machine Management
13. Battery Life
14. Mac Pro: Gaming
15. Mac Pro: CPU
16. Conclusions
[/one_half_last]