Beyond vmkfstools: alternatives to VMware's backup tools

VMware includes decent tools to backup virtual machines. But there are more poweful alternatives - some free - that deserve your attention.

There are many ways that you can back up virtual machines on VMware ESX hosts. You can use traditional tape backup methods with agents inside the guest operating system, VMware Consolidated Backup in conjunction with third-party backup applications, or you can use specialized virtualization backup applications like PHD Virtual Technologies esXpress, Veeam Backup and Vizioncore vRanger Pro.

However, there are times when you need to quickly make a backup of a virtual disk. In this tip, we will cover alternative methods that you might utilize in addition to your primary backup methods. These backup methods work by either copying the large virtual machine disk format (.vmdk) file to another disk storage device, or by cloning a virtual machine to another host or storage destination. These types of backups are useful for doing a bare-metal restoration of a virtual machine, and can also be used to restore individual files.

SCP and vmkfstools

Let's begin with the simplest method that involves making a copy of the virtual disk of a VM using a utility such as a Secure Copy Protocol (SCP) utility or the ESX command-line utility called vmkfstools. The difference between the two is that vmkfstools is specifically designed to work with VMFS volumes and can only be used for copying a virtual disk between either local or shared VMFS volumes, and SCP can be used to copy a virtual disk to any type of storage device. Additionally, when copying with vmkfstools, all space is allocated at once so less fragmentation occurs. SCP, on the other hand, allocates space as the copy progresses, which can cause virtual disk fragmentation. For this reason, vmkfstools is the preferred utility when copying virtual disk files between or to VMFS volumes. SCP, however, can be used to copy virtual disks from VMFS volumes onto other storage devices and also back to VMFS volumes. Once you copy a virtual disk file back to a VMFS volume using SCP though, it's a best practice to use vmkfstools to make another copy of it first to avoid fragmentation. But if it's only for temporary purposes, this isn't necessary.

If you are going to copy a VM's virtual disk while it is powered on, you should always create a snapshot of the virtual machine first so the disk becomes read-only and can be copied safely. You can create the snapshot using the VI Client or by using the VMware-cmd command line utility. While the copy operation is completing, all writes to the disk will be made in a separate snapshot delta file. Once you complete the copy you can delete the snapshot, which will write the changes from the delta file back to the original disk file.

For copying using SCP, you can use the command line secure file copy protocol utility built into the ESX Service Console, but it is easier to instead use one of the third-party free utilities like Veeam FastSCP or WinSCP for this. This is a manual copy method that is good when you need to make ad hoc backups of your virtual machines before applying patches or upgrades to it. However, there are ways to automate this method. Veeam's FastSCP comes with a built-in scheduler to allow you to schedule copy jobs to run at certain times. Before you do this, you need to have a destination disk for your copy such as a local disk on a workstation/server or a mapped network drive on a server. In the example below, we are using FastSCP to copy a virtual disk from an ESX host to the local disk of a PC workstation.

1. If you do not already have FastSCP you can download it for free and install it.

2. If you have any existing snapshots, it's best to delete them before making a copy of the virtual disk. Once you are ready, either power off the virtual machine or create a new snapshot of it so the virtual disk becomes read only.

3. Run FastSCP, select File, Add Server and add your source ESX host or specify a vCenter Server to add all hosts that are managed by it. Complete the Add Server wizard and click Finish.

4. Once you connect to the host you have added you can browse its filesystem, including all of the VMFS volumes that it has access to. Browse until you find the directory of the VM that you want to back up. If you want to back up just the virtual disk file, then you need to copy both the large –flat.vmdk file and also the smaller disk descriptor file. Both of these files should have the same name with the exception of the –flat part of the larger file. Don't copy any vmdk files that have –delta or -0000001 in the file name as those are snapshots. You can also copy all of the smaller files, but they are not necessary to recover the VM as the actual data of the VM is in the vmdk files.and select the Copy option (see below).

5. In the left pane, select either another ESX host or a local/network drive under My Computer, then in the right pane right-click and select Paste as shown below.


6. The copy will commence as shown below. Once it completes, you will have a backup copy of your virtual disk and you can power the virtual machine back on or remove the snapshot.

As mentioned earlier, you can also use the vmkfstools Service Console utility to make backup copies of virtual disks on to VMFS volumes. This is a special utility that VMware has developed for virtual disk file operations and is designed to work with VMware's proprietary VMFS volumes. When used with the –i parameter it will create a cloned copy of your source virtual disk file to whatever directory you specify on a VMFS volume. You can also specify to split the disk file into separate 2 GB files using the –d 2gbsparse parameter (2 GB Sparse format) that can make it easier to copy to other systems or to put onto removable media like a DVD-ROM or flash disk. But you can't power on a virtual machine in this format on an ESX host without first importing the virtual disk into in a thick or thin disk format using vmkfstools. To create a backup copy of a virtual disk, follow these steps:

1. Login to the ESX Service Console.

2. If you have any existing snapshots, it's best to delete them before making a copy of the virtual disk. Once you are ready, either power off the VM or create a new snapshot of it so the virtual disk becomes read only.

3. Switch to the directory on the VMFS volume of the VM that you want to create a backup copy of.

4. You can type vmkfstools by itself to see the options and command syntax, additionally documentation on using it can be found here. The syntax for creating a clone of a virtual disk is vmkfstools –i . The destination disk name can be any directory on the same or separate VMFS volume. Note: Because vmkfstools was written for virtual disk operations, it is aware of the two files that make up a virtual disk, and subsequently you only need to specify the smaller non-flat file name. The clone operation will proceed.

5. Once the clone operation completes, you will see a new disk file created that will be an exact backup of the source virtual disk and you can power the VM back on or remove the snapshot.

There is one final method that you can use to make backup copies of virtual disk files by using the Datastore Browser built into the VI Client. Beginning with ESX 3.5, the Datastore Browser was enhanced to include operations like copying, renaming and deleting files on VMFS volumes. If you select an ESX host and choose the summary tab, you can right-click on a datastore that is listed and choose the Browse Datastore option. You can also switch the Inventory view in the VI Client to Datastores to see a listing of all datastores. To use the Datastore Browser to make a backup copy of a virtual disk follow these steps:

1. Launch the VI Client and connect to an ESX host or vCenter Server.

2. If you have any existing snapshots, it's best to delete them before making a copy of the virtual disk. Once you are ready, either power off the VM or create a new snapshot of it so the virtual disk becomes read only. If you don't, you will receive an error message if you try to copy the disk.

3. Right-click on the datastore that your source VM is located on and choose the Browse Datastore option.

4. You will see a listing of VM directories; doubleclick on the VM that you want to back up and you will see a listing of all the files in that directory. The Datastore Browser is a bit different as it displays virtual disk files in a friendly format and only shows one file for each virtual disk even though two exist. If you right-click on a virtual disk you will see various options including Download and Copy as shown below. The Download option will allow you to download the virtual disk to either a local or network drive on your workstation. The Copy option will allow you to copy the virtual disk so you can paste it to another directory on the same VMFS volume or to a different VMFS volume.

5. If you want to copy it to another VMFS volume, just close the Datastore Browser after you select the Copy option. Then, open it again on another VMFS volume, and browse to the directory that you want and right-click and choose the Paste option as shown below. Also note a limitation of the Datastore Browser is that it will not let you Paste in the same directory as the source virtual disk file is located in.

6. Once the operation completes you will have a new copy of the disk file and you can power the VM back on or remove the snapshot.

So now that you have made a backup copy of your virtual disk what do you do if you need to restore it or access files on it? There are several ways to do this:


  • If you wish to do a bare-metal (full) restore of the virtual machine, you can simply shutdown the VM, copy the virtual disk back to the VMFS volume and remove the current hard disk from it and add a new disk and select to use an existing disk and browse to the virtual disk file that you restored. You can also create a new VM instead, and tell it to use the existing disk that you restored and then delete the original VM afterwards.

  • If you wish to restore individual files, the easiest way is to copy the backup copy of the virtual disk back to the VMFS volume if its not already there and then add it as an additional hard disk to either the original VM or another VM and then access the newly added disk and copy the files that need to be restored back to their original locations. You could also create a new VM and configure it to use an existing disk, and then power it on and copy off the files that need to be restored.

In addition to these methods, there are other methods that you can use to clone a virtual machine for backup purposes by using the vCenter Server clone function and task scheduler, as well as using the vCenter Converter application. Having the flexibility of multiple backup options to use depending on the situation is one of the great advantages of a virtualized environment. Making quick backup copies of a whole virtual disk is good insurance especially when doing potentially risky operations like increasing the size of a virtual disk or operating system partition.

Read more on Data protection, backup and archiving