Overcoming VMware vShield Endpoint kinks and challenges

VShield Endpoint improves VDI performance by offloading anti-virus processing from guest VMs, but can be tricky to install and manage.

VMware vShield Endpoint is an agentless anti-virus tool for VDI environments that provides a lot of advantages over traditional approaches to desktop security. Unfortunately, IT pros also find it tricky to deploy and manage.

It is however possible to improve the outcomes for a vShield Endpoint deployment by making a few simple changes such as updating VMware Tools, customising the installation process; and eventually writing some custom scripts.

vShield Endpoint explained

As one of the components of the VMware vShield security suite, vShield Endpoint offloads the antivirus (AV) and anti-malware functions from a guest operating system to a dedicated virtual security appliance. Targeted at vCloud Director and VMware View environments, it is useful in cases when server or desktop consolidation causes excessive CPU and disk activity.

VShield Endpoint works by loading a driver inside a guest operating system (OS) that is linked to a hardened VM tasked with looking for viruses or malware on the guest VMs.

This feature – relocating antivirus tasks away from guest OS – is a welcome change from the traditional approach where the antivirus software is installed to the guest OS and uses computing resources each time it is executed.

IT professionals have always longed for a method to carry out security tasks without using agents. Agents are a hassle: they consume resources, they require endless maintenance and updates for every instance on the network; and licensing them is problematic.

As such, vShield Endpoint seems like a technology whose time has come.

Using vShield Endpoint means updating VMware Tools
I recently used vShield Endpoint alongside a VMware View deployment. It had become obvious that using traditional mismanaged AV agents would gobble up valuable CPU cycles, degrade the user experience and affect scalability. By relocating antivirus outside of the guest OS, Endpoint simplified security management and made scanning and quarantining more efficient.

But installing and configuring vShield Endpoint wasn’t without its challenges.

For instance, with vShield 5.0 suite, VMware no longer includes by default the SCSI driver that facilitates communication between the protected VM and the vShield appliances themselves. But vShield Endpoint system does need a driver from VMware Tools to install correctly.

One way admins can overcome this challenge is by updating their VMware Tools.

If you have used the “Complete” option when deploying VMware Tools, and have kept VMware Tools up-to-date, you’re in a good position. But if you haven’t, then you should update VMware Tools to include the vShield Endpoint driver.

Admins updating VMware Tools manually can find the vShield Endpoint driver under the “VMCI” folder in a “Custom” installation mode.

Custom setup in vShields Driver
Caption: By default, a typical installation does not include the vShield Drivers held under the “VMCI Driver” section of the VMware Tools install. The drivers are required on the endpoint, and offer improved performance

A simple way to ensure the driver is installed to every virtual desktop would be to modify the “Parent VM” that makes up a VMware View “Linked Clone” and then re-create the virtual desktops based on this image.

Whilst considering a VMware Tools update, admins should also check with their third-party AV provider if they have a “silent agent” or “thin agent” that can be installed at the same time.

A silent agent is an optional component which gives end users an interface from which they can be notified of possible virus infection, and decide whether to delete, fix, or quarantine the infection. 

Thin and silent agents aren’t mandatory but admins should consider them as they will help them address two issues. For one, these agents have a payload that is a fraction of a full-blown AV software installation. For another, the tray icons and messages that these agents present help reassure users that their desktops are safe.

BitDefender “Silent Agent” installed to Windows 7
Caption: This screen grab shows the BitDefender “Silent Agent” installed to Windows 7.

Minimising vShield Endpoint overhead
Another VMware vShield Endpoint challenge is the payload that the vShield suite itself generates. Deploying vShield Endpoint in a four-node ESX cluster requires a total of six virtual appliances–  two for management, and another four across each ESX host called Security Virtual Appliances or SVAs. These SVAs are the work horses of the system as they are the systems which actually check for infections to the protected VMs.

The first management appliance is the vShield Manager, and then second is the Security Management Console provided by the chosen antivirus vendor. The vShield Manager offers APIs that can be used by the third-party Security Management Console, while the Security Management Console deploys a virtual appliance to each one of the four ESX hosts. These latter appliances are responsible for protecting the VMs on the ESX hosts.
vShield Endpoint implementation for a four-node ESX host cluster

Caption: For a four-node ESX host cluster a vShield Endpoint implementation would result in six virtual appliances - the core vShield Manager, the third-party AV provider Security Management Console as well as SVA for each host.  Remember the management consoles are installed once per vCenter instance.

In theory, the memory, network, disk and CPU resources demanded collectively by each of the appliances should add up to less than the resources consumed by running traditional AV agents inside each VM – provided the ratio of VMs to ESX host are sufficiently high.

But one Security Virtual Appliance protecting two VMs on an ESX host hardly stacks up from a resource or licensing perspective. Additionally, admins may want to consider the “management overhead” caused by having to configure each of the VAs that allows the tool to run.

To simplify matters, most vendors have tools to remotely push the virtual appliances to each ESX host in the cluster, handling simple issues such as host naming conventions.

However, each virtual appliance needs its own IP address. In my case, I was able to deploy the appliance using DHCP, as this is supported on the server side of the network. Sadly, that’s not an option for all environments because many don’t regard DHCP on the server network as secure.

Of course, I had to consider that managing a number of virtual appliances is easier than managing all the agents in each and every VM.

Beware vMotion and vSwitches
Finally, one last tip to IT pros using vShield Endpoint for the first time: Watch out for vMotion and features and functions that depend on it. By default, the VSA is configured to communicate to an internal vSwitch that is automatically created on every ESX host during the deployment phase. This secures the communications between the appliances, the host and vShield-protected VMs. Once a VM or virtual appliance is configured for such a vSwitch, it is no longer “vMotionable” and this can have a trickle-down effect on other administrative tasks.

standard switch

The most significant effect is when the VMware Update Manager (VUM) triggers a “maintenance mode” as part of the process of patching an ESX host. This works most efficiently with Distributed Resource Scheduler (DRS) in “fully-automated” mode, which moves VMs off to other ESX hosts in the cluster until there are no more VMs on the original host. VUM then installs the patches and reboots the host before re-joining it to the cluster, and the whole process begins again with the next host.

However, in case of a DRS Cluster running vShield, maintenance mode gets stuck at 2% completion, as it has no capacity to move the SVA.

It would be helpful if VUM had some method of excluding virtual appliances like the vShield SVA, or at least suspend and resume it once the update process was complete. For now, admins will have to consider scripting this with tools such as PowerCLI.

Overall, my experience deploying VMware vShield Endpoint has been positive. While it is not without its kinks and challenges, such tools will eventually become the de facto standard for antivirus in a fully-virtualised enterprise.


Mike LaverickMike Laverick is a former VMware instructor with 17 years of experience in technologies such as Novell, Windows, Citrix and VMware. Since 2003, he has been involved with the VMware community. Laverick is a VMware forum moderator and member of the London VMware User Group. He is also the man behind the virtualisation website and blog RTFM Education, where he publishes free guides and utilities for VMware customers. Laverick received the VMware vExpert award in 2009, 2010 and 2011.

Since joining TechTarget as a contributor, Laverick has also found the time to run a weekly podcast called the Chinwag and the Vendorwag. He helped found the Irish and Scottish VMware user groups and now speaks regularly at larger regional events organised by the global VMware user group in North America, EMEA and APAC. Laverick published books on VMware Virtual Infrastructure 3, vSphere4, Site Recovery Manager and View.

Read more on Virtualisation management strategy