More lost virtual machines were caused by problems in snapshots than by faults in their underlying hardware. An operating virtual machine is composed of links along a chain. This makes each of them vulnerable to failure, where if one link is broken, an otherwise perfect base disk for the original machine cannot be used. What follows is an explanation of how that happens and how it is put back.
Consolidation of the disks causes changes to occur within the links of each chain; however, on a damaged datastore, consolidating can create damage to the very links that a recovery process will have to rebuild. If your virtual machine fails to boot and there is suspicion about the storage device, do not move or alter your snapshots.
More virtual machine failures result from problems created in snapshots than from any hardware failure. They deserve your study prior to your having to use them.
Snapshots do not replicate virtual machines. Instead, at the moment a snapshot is taken, all subsequent writes made to the base disk are recorded into a new delta file. Then additional deltas are written on top of previous ones. So while the entire run-time virtual machine is comprised of a series of chained files, and therefore all of these files together represent the current state of the machine, removing a single link in the chain will stop the running machine, regardless of whether or not its base disk has no errors.
Two bad things happen. Snapshots may have been running for months, so they are now so big that the datastore has filled completely and all operations against it stop. Consolidation; merging the deltas back down into a single file is a very large write operation and can be broken halfway through leaving a series of files that are neither consolidated nor consistent. Doing this on a datastore that is currently experiencing failures is how a recoverable VM becomes a difficult one.
Then you can't do the second one until you can read the first one, which is why these jobs run sequentially.
In addition to environments people didn't document prior to when their creator left.
VMware ESXi and vSphere — VMFS datastores; VMDK flat files & delta files; snapshot chains; thin-provisioned disks; also datastores presented from a SAN that must be rebuilt first. Microsoft Hyper-V — VHD & VHDX; differencing disks; checkpoints; Cluster Shared Volumes.
Proxmox, XenServer and KVM — QCOW2 and raw images, LVM-backed guests and ZFS datasets. Recovered machines return to either bootable disk image(s) that can be attached immediately, or extracted file(s), depending on your preference for rebuilding the VM clean. We'll find out which method works best prior to completing.
Priced higher than array-based recovery since the recovery will continue beyond the datastore.
We price virtual machine recovery from £800 +VAT (confirmed via a complimentary 48-hour diagnostic). If the layer below requires reconstruction to recover the virtual machine, then that cost will also need to be quoted. That may include RAID and/or server recovery from £500 +VAT, enterprise SAN recovery from £1,250 +VAT. In cases where physical drive level work is required, an initial 50% deposit will apply.
Please let us know the name of the hypervisor and its version number; how many machines require recovery; if snapshots are in place; and all attempts made so far — especially any consolidations. Non-disclosure agreements are standard and invoicing is available.
Provide the virtual disk files. Tell us which hypervisor you are using. Provide the file format. Tell us what went wrong. This diagnostic will cost you nothing. One price quote must be given in writing before we begin any work.
Nothing starts until the virtual disk files — or the failed storage device containing those virtual disk files — reaches our bench. Properly package it and include your contact information with it. Provide the hypervisor used and the format of the disks. There is no charge for diagnostics, and prior to beginning any work we will provide you with one written estimate.
Posting it? Use a tracked and insured service. The value of what is on the drive far exceeds the cost of postage. Bringing it in? Weekdays from 9am to 5:30pm, and it still needs packing as described above for the journey.
If you're unsure if it's worth sending then tell us about the hypervisor, disk format and fault, along with what you have done up to now – an engineer will read each of these and give their honest assessment of the odds and a cost range before you send it out.
We’ll be in touch shortly. If it’s urgent, call 0161 871 0788.
These are the types of questions we're asked the most about recovering your virtual machines.
If no other data has been written since then; yes, it frequently is. Delta files orphaned from their parents can be rebuilt into the proper order. The only thing that would make it unrecoverable is if you ran a consolidation on a datastore that was already failing.
A snapshot records all changes to a continually expanding delta file instead of recording to the base disk. When left alone for months, these grow until they fill up the entire datastore. And when they do, all machines stop working immediately because they cannot write anything else.
Usually. If an array failure exists, we will image and reconstruct it first; then rebuild the VMFS datastore to make your virtual disks accessible again. All VMs on that datastore will be recovered at one time.
Yes. Once the virtual disk is readable we treat the guest file system as with any other. Files deleted within the VM, damaged guest file systems and corrupted databases are all recoverable in the usual way.
You can choose to either receive a bootable disk image of the VM that you can attach directly to a server or you can request that your files be extracted from the VM and built back onto a new clean machine. We will contact you prior to completing our service to determine which option best meets your needs.
From £800 plus VAT after a 48-hour complimentary diagnostic. The price for rebuilding the array or SAN required to restore access to your data will be included in the quote. This will vary depending upon what type of array or SAN was used, but examples include: RAID/servers from £500 + VAT, enterprise SAN from £1,250 +VAT.
A clean copy of a virtual disk does not necessarily mean it will start, therefore they are tested as guest VMs instead of testing the file itself. Please provide the hypervisor, its version and most importantly whether a snapshot consolidation has already been performed.
From £800 +VAT. Tell us the hypervisor and version, number of guests affected and whether you have used snapshots – as well as whether a consolidation has already taken place, since that alters what can be worked with.