Burton Group report says VMware beats Hyper-V and Xen for hypervisor performance

Analyst firm Burton Group has staged a showdown between four hypervisors and found all lack some features, but that VMware emerges on top of the heap.

IT professionals have an array of hypervisors to choose from these days, all with their own pros and cons. So how do you make the right choice?

To help IT select the right hypervisor for its environment, the Burton Group VMworld Europe in Cannes, France, this week.

In addition to providing a hypervisor scorecard, the criteria in the study should be administered by IT as a way of differentiating between true hypervisor features and marketing spin, Wolf said.

"We believe this is the first comprehensive hypervisor comparison study by an analyst firm, and some people – like Microsoft – are not going to be happy with us," Wolf said. Burton Group worked with Global 1000 organizations on the evaluation criteria and analysis of production level hypervisors.

The evaluation criteria
Burton Group scored four products that it deemed production-environment ready: VMware Infrastructure 3 (VI3), Microsoft Hyper-V, Citrix XenServer and Virtual Iron and scored each hypervisor using several features, including the following:

  • high availability;
  • live migration;
  • memory management;
  • networking, storage and security;
  • management features and power; and
  • licensing and support.

Each hypervisor was scored on the sophistication of these features and whether the hypervisor has the required components as well as the level of "preferred" and "optional" features included.

When it comes to high availability, a hypervisor has to eliminate the physical host as a single point of failure by having the ability to assign virtual machine (VM) priority levels for restart. Some preferred features including guest OS and application failure detection and partial physical node failure detection, and an optional bonus is integration with third party HA software like Symantec, according to Wolf.

Live migration is extremely important for users, and Burton Group looked at this along with support for live migration across different platforms and the capability to simultaneously live migrate two or more VMs. VMware, Virtual Iron and Citrix all offer live migration, and Microsoft plans to offer this in its next version of Hyper-V , but until then, many users cite this as a reason not to deploy Hyper-V.

Support for hardware-assisted memory virtualization - AMD's Rapid Virtualization Indexing and Intel's Extended Page Tables – was also considered because these hardware features improve performance significantly for many applications, according to Wolf. Memory overcommit and large page table support in the VM guest and hypervisor are preferred features, and memory page sharing is an optional bonus feature, Wolf reported.

In networking, the Burton Group required that hypervisors support network interface cards (NICs) teaming and load balancing, Unicast isolation and support for the standard (802.1Q) virtual local area network (VLAN) trunking.

Each hypervisor also had to support iSCSI- and Fibre Channel-networked storage and enterprise data protection software support, with some preferences for tools and APIs, Fibre Channel over Ethernet (FCoE) and virtual disk multi-hypervisor compatibility.

Management features including Simple Network Management Protocol (SNMP) trap capabilities, integration with other management software and fault tolerance of the management server were also deemed invaluable to a hypervisor.

The hypervisor scorecard
VMware VI 3.5 Update 3 satisfied all Burton Group's criteria 100% and outshone the competition but is still missing some features, according to Wolf. The product would be better if it included partial physical node failure detection, dynamic I/O buffering and cache tuning, bi-directional Challenge Handshake Application Protocol (CHAP) Internet Storage Name Service (iSNS) protocols, and Trusted Platform Module (TPM) support. It also lacks virtual hard disk compatibility.

Microsoft Windows Server 2008 Hyper-V satisfied just more than 80% of the "required" criteria, but it lacks important features such as high availability and live migration. It also doesn't allow users to prioritize VMs for restart following a failure, doesn't include hardware-assist memory virtualization support, or fault tolerance, Wolf reported.

When it comes to satisfying the criteria, Citrix XenServer 5.0 scored about the same as Hyper-V but has features that Hyper-V lacks (including live migration and fault tolerance). XenServer's Achilles heels are in the areas of security and networking, according to Wolf. XenServer 5 is missing 802.1Q virtual local area network (VLAN) trunking, and as far as security it does not offer directory services integration, role-based access controls, or security logging and auditing or administrative actions.

Virtual Iron also scored just more than 80% in satisfying criteria, but poorly in the area of preferred and optional features. What's missing? An enterprise-class support policy and some management features - fault tolerance, third-party integration and SNMP, according to Wolf's VMworld Europe presentation.

The fact that Microsoft has yet to release a full-bodied hypervisor comes by no surprise to enterprise users, said Frank Lacomba, the director of product management at SunGard Availability Services, which supports customer virtualization deployments. "Enterprises typically wait until the first service pack, when they get their act together," before adopting Microsoft software, Lacomba said.

"Adoption by IT is based on maturity, and VMware is mature and feature rich - they are on version 3.5 now, and the product is very stable," Lacomba said. "We see VMware as the dominant leader, and they probably will be for the next 18 months, at which point Microsoft might be able to challenge them if they prove that they are cheaper, faster and better."

SunGard supports customer virtualization deployments using VMware, Solaris, and virtualization on AIX and AS/400. It is working on the certifications to support Hyper-V.

Gentry Ganote, a former CIO who has his own virtualization and storage consultancy firm called Rojoli Technologies in Atlanta, uses both VMware and Virtual Iron, and sees a use case for both.

" Virtual Iron has a particular market segment … it is good for a small shop, but for the big guys with 10, 20, 30 compute nodes, VMware really shines," Ganote said. By comparison, he said "VMware is pretty bad-ass; they charge a lot for it, but it is sophisticated."

It is understandable that a small company like Virtual Iron wouldn't have robust third-party integration, because they are not the market leader, Ganote said. Other tradeoffs, like the lack of fault tolerance, can be justified by a much lower price, he said.

Read more on Virtualisation management strategy