Can I Bench With Virtualization?

posted Monday, 7 April 2008

Can I run a benchmark using virtualization is a question that I've been asked on a number of occasions. The short answer is, no, you can't. The first hint that virtualization wasn't going to be an option was when I first installed VMWare on my Compaq/HP NC6000 laptop. The whole experience sucked. It corrupted my email and did a whole lot of other nasty things to my machine. After a bit of poking about I attributed it to interference with the network drivers. It was clear to me that network performance, arguably one of the most important considerations in any benchmark, was woefully inadequate.

Too bad because other than that, the benefits of virtualization were clear. I attributed to a bad hardware experience and tried to uninstall. Unfortunately, VMWare didn't uninstall so nicely and it took a re-install of windows to get rid of it. With that experience in hand, I was very reluctant to try out VMWare on my Mac. I've installed it and though it is more useful and doesn't seem to corrupt things, you can tell that performance is an issue.

Just today this popped up in my inbox. Queuenews reports;

The efficiency of network virtualization directly impacts the number of
network servers that can effectively be consolidated onto a single
physical machine. Unfortunately, modern network virtualization
techniques incur significant overhead, which limits the achievable
network performance. We need new network virtualization techniques to
realize the full benefits of virtualization in network-intensive
domains.

Not a surprise from my point of view but just more fodder to say, virutalization... cool but not for performance just yet.

comments (6)

1. Bram Somers left...

Monday, 7 April 2008 8:04 pm

Why not installing more than one physical networkadapter on a server machine? And let VMWare (or any other virtualization software) assign guest systems to different physical hardware? I agree to your opinion that performance CAN be a problem for virtualisation, I don't agree that it is ALWAYS the case, tuning is possible.

Greets,

Bram

2. William Louth left...

Tuesday, 8 April 2008 4:39 pm :: http://blog.jinspired.com

You can benchmark if this is the target platform. Many companies are using virtualization for more things than just demoing or testing an application.

3. Jozef Babjak left...

Friday, 25 April 2008 3:11 pm

Unfortunately, my experiences with heavy-weight virtualizations like VMWare are similar. However, there are virtualization solutions which can be used also for a performance testing. Namely, BSD jails and - maybe more familiar with a Java world - Solaris zones. If carefully designed (nothing in global zone/jail, pararell zone/jail running only for setup, but dedicated running during benchmarking, and so on), these are suitable for benchmarking.

4. Kirk Pepperdine left...

Friday, 25 April 2008 3:22 pm

I'm a skeptical.

5. Anindo Banerjea, TBD Networks left...

Thursday, 5 June 2008 9:57 pm :: http://www.networkvirtualizationblog.com

I was trying to do some networking performance tests and ran into the same issues to start with.

You might find this white paper from VMware useful.

http://www.vmware.com/files/pdf/ESX_networking_performance.pdf

The bottom line seems to be that you can get the network performance up to comparable to native hardware, but you have to install the vmxnet driver. Also, you have to enable some things (Jumbo frames, TCP Segmentation Offloading). Then you can get the performance up there with native hardware.

One caveat is that this paper is only about ESX. So does not apply to your laptop scenario. Which version of vmware was your experience?

6. Kirk Pepperdine left...

Friday, 6 June 2008 5:44 am

Fusion 1.0. But I've also experienced worse using other versions running on Windows.

That said, I'm a little concerned about the extra ordinary steps one has to take to get the bench to work. I would be concerned that the bench would be affected even so.

Even with this information I still stand by my conclusion; as much as I wished it worked, it doesn't. I do hope they get the performance bugs shaken out because virtualization would be very useful. In fact even with the performance bugs, it still is useful. Just not for benching (or prod) IMHO.