In the event that something happens to a Xen virtual machine (VM) that prevents you from starting it, it's a good practice to have the virtual machine storage back end mounted in the Linux file system of the Xen Server physical machine. By doing so, you'll be able to repair the VM quickly and painlessly. In this tip, I'll cover how to do this for physical devices that are used as storage back ends.
Why should VMs be mounted on physical machines?
Xen uses a virtual hard drive (VHD). This virtual hard drive is stored somewhere on the Xen Server host machine. For the purposes of this tip, I will refer to this "somewhere" as the storage back end. There are two common ways to configure the storage back end. The easiest solution is to use a disk image file which is created in the host operating system. This also is the location where the VM installs its files. The alternative to this method is to use a device as the storage back end. This can be a logical volume, a partition or a logical unit number (LUN) on the storage area network (SAN). Essentially you can use any storage device for this purpose, and due to the performance benefits this solution is used most often.
By mounting a virtual machine in the host operating system (OS), you can do all maintenance that you need to do rather easily. If the virtual machine runs Linux as its operating system, this is particularly easy because the Xen host OS is Linux as well. It also helps that the most useful tools for VM maintenance in a Xen-based environment are installed by default in the Linux host. But even if the virtual machine uses Windows there are lots of options. You use one of the many new technology file system (NTFS) maintenance tools that are available nowadays.
If the tools you need are not present in the host operating system (which may be the case for some of the advanced NTFS tools), you may need to reboot the physical machine from a Linux live CD such as Knoppix. Please note that if you do this, don't forget to migrate all other VMs that are still available on your Xen server to another physical server in the network.
Mounting a physical storage back end
To be able to mount a storage back end, it's important that you first determine what it is that you want to mount. This is because the procedure for physical devices is different than the one for disk image files. I'll explain the procedure used for Xen image files in a future article.
Let's assume though that the storage back end device is visible on the host machine as the logical volume manager (LVM) with the name /dev/hosts/vm1. If you are using something else (a partition, a complete hard disk or a LUN on the SAN), this approach also works for you. All you have to do is replace the name /dev/hosts/vm1 by the correct name of the device that you are using.
From the perspective of the Xen virtual machine, the physical storage back end contains a complete hard disk. That means that there is also a partition table inside, and in order to mount the file systems in the storage back end one-by-one you first have to find out how the storage back end is partitioned. You can do this by using the following command:
fdisk -l /dev/host/vm1
Now you know what partitions exist within the storage device. However, you can't mount the partitions just yet because the host OS doesn't recognize them and you don't have a device file present in the host OS. Fortunately the kpartx tool, which is part of the multi-path tools package, allows you to create these device files. You can utilize this tool by using the following command:
kpartx -a /dev/host/vm1
At this point you will see all of the required device files in the /dev/mapper directory. These device files typically start with the name of the device that you've used, followed by a partition number for the particular device. For instance, the first partition from /dev/host/vm1 will become /dev/host/vm1p1.
Next, you need to find out which of these partitions contains the root file system. Once you've discovered which partition has it, you can mount this partition onto the physical machine. If you're not sure which partition contains the root file system, just mount the partitions one at a time and try them all. The example command below assumes that the root file system is in partition 2 on vm1:
mount /dev/mapper/vm1p2 /mnt
At this point you can access all of the files in the storage back end of the virtual machine. As a result, all modifications that you make to your VMs will be stored to these files as you now have full file system access. Remember though to un-mount the partition and use the kpartx tool to remove the temporary device that you've created. The following two commands will do that for you:
kpartx -s /dev/host/vm1
There is no need to invest in expensive tools to troubleshoot a virtual machine. You can just mount the virtual machine storage back end. In this tip you've learned how to do that if the storage back end is located in a device.