Software RAID for Linux at the base, an LVM thin pool over that, then the file system and finally the share. Errors usually appear at a higher layer than the actual one that has caused a failure, hence the layer where you see an error may not be the actual layer of failure.
The QNAP typically uses a combination of thin provisioning and a storage pool on top of its RAID. Therefore, when an entire volume fails, all disks under it could still function normally. That’s why recovery means unwinding these layers in order.
This explanation can help understand most QNAP failure modes.
Assembling your disks together using standard Linux RAID. It is common and easily understandable. This is the reason why recovering from a QNAP failure is generally easier compared to a proprietary hardware controller.
The array presents itself as a physical volume and it is divided up into logical volumes. Typically for QNAP, there is a thin pool involved, which is where issues arise.
A conventional file system sitting atop a logical volume. It has "superblock" copies that a recovery process can rebuild from.
This is what users experience when accessing data over the network. If an error occurs here, it appears indistinguishable from a storage failure from a client workstation. This is why we have to check whether ping works and whether the administrative interface shows any signs of life before concluding that the worst possible outcome occurred.
The biggest difference in recovering from a QNAP versus a standard array.
Thin provisioning provides an available size for a volume, however no actual reserves have been made. In other words you can create a 10 TB volume in a 6 TB pool. Available space will be assigned to the volume based upon the amount of data being written. Thin provisioning is very efficient and it is also the source of a failure mode that surprises people.
When the pool reaches its maximum usage write operations will fail. However the volume may report there is available space, since it was never supported by the storage that was advertised. Once again, the file system believes it has available space to continue writing, leaving behind partially completed structures. QNAP devices provide alerts regarding the utilisation of your pool. However these warning messages are commonly ignored as part of your overall capacity plan. They should be viewed as a warning related to the potential instability of your device.
In terms of recovery, both the mapping metadata and the data itself matter equally. With a thin pool, each physical extent maps to a logical address. To successfully recover a volume, it is necessary to properly walk through this mapping. If not done so, a recovered volume will appear as random noise even though all disks are readable.
The unit runs in a degraded state until another disk fails. This is the most common failure to reach us by a large margin. Preventative measures include setting up alerting that allows someone to notice that the failure occurred.
File system has been left in an inconsistent state with reported available free space due to thin volume over-commitment and mid-transaction write failure.
Unit boots up and can’t mount its own volumes. Typically the disks remain untouched. Issue lies with the appliance.
Logical damage occurred to ext4 structures mid-write. Damaged structure was repaired via backups of the file system's own superblock(s).
Appliance has been taken completely out of the loop.
Firstly, every single disk is run through a read-only imaging process. Any disks the unit has previously ejected will be included in this process. Once completed, the mdadm array can be reconstructed from these disk images using the geometry found within those images; not the geometry provided by the device that failed to assemble it. Each piece of information is tested to find the correct parameters to provide for coherent assembly: stripe size, member order and parity rotation.
LVM metadata is then extracted to determine how the data was laid out and to identify the mapping from logical address to physical extent. On a thin pool, this allows the location of the actual data to be identified. Finally, the ext4 file system appears and can be repaired from its own backup superblock(s), should they be required.
Because every level of abstraction is standard Linux based storage, none of this requires either QNAP's firmware to be active or for the unit to have powered up. After a complimentary 48-hour diagnosis, we charge from £500 +VAT to recover your NAS and/or RAID system in Manchester. We accept either disks labelled per bay or the entire unit — whichever you prefer.
Less than most people would anticipate. Beneath its user-friendly surface the appliance uses conventional Linux storage — mdadm, LVM, and ext4 — so when the physical disks have been successfully imaged, reconstruction is normal and the appliance itself is no longer relevant.
Thin volumes report a size they have never allocated; instead, they allocate space only as data is written to them. If the underlying pool reaches capacity, then disk writes will fail even though the reported remaining free space in the thin volume appears available — leaving the file system with partially completed structures.
Generally not. Most often that's just typical ext4 damage due to an interrupted write operation that is traditional logical damage that is repaired by rebuilding using the file system's own superblock backups. Usually the disks themselves remain intact.
Yes. Because all layers are utilising standard Linux storage, you can mount the volume and assemble the array completely independent of the appliance. One of the better scenarios for a dead QNAP appliance would be if it had disks that were good.
First copy all of your information to another device. The rebuild process will read every sector on every existing disk in the array for hours. This is the highest amount of I/O (input/output) the entire array experiences. It is also applied to disks that are the same age as the disk that has already failed.
From £500 plus VAT. After you receive your free 48-hour diagnostic, we will require a 50% deposit if a member requires physical drive-level work. Please send the labelled disks from each bay or the whole unit back to us.