The quiet first failure
When a member goes down nobody knows until it happens again — the array continues to serve as long as there are enough working members to do so. Weeks or months go by before someone reads the alarm.
Most common
Distributed parity in a RAID 5 means that if any one of the members fails during an operation, then the RAID will still be able to serve. And that is exactly why we tend to see arrays that have already lost a second member mid-rebuild by the time they get here. Below we outline what type of calculation is possible when using parity to recreate information, what calculations are not possible with parity and how the very act of rebuilding the array is the riskiest period for the array.
Because of parity — any individual member of an array may fail and you can calculate the missing data from the other members. When the second failure happens — the maths stops; you now need to physically get one of the failed disks readable again so that you can continue with the mathematical recalculation of the data that was originally stored.
How a RAID 5 continues to operate even though it has a member that is no longer functioning — and why a RAID 5 cannot continue operating once it has had another failure.
Just as a RAID 0 divides the storage across all the drives in an array, a RAID 5 divides data across each drive within an array; however, a RAID 5 adds a "parity" block, which is calculated as the "XOR" of the other blocks within that same stripe. This parity duty is rotated throughout the entire set of disks, stripe by stripe. If any one member of a RAID 5 goes down — every missing block can be generated dynamically from the remaining blocks in its stripe. Thus, the array remains operational although at a lower performance rate and without protection.
The capacity maths gives an entire disk's worth to parity — four 4TB disks create a 12 TB volume. The protection maths is tighter. Only one unknown per stripe can be solved; two cannot. If a second failure occurs in the array, the array does not just degrade further, it stops.
Almost all RAID 5 arrays which reach our bench reached it after failing with the second failure — and usually while rebuilding the array from the loss of the first failure.
Rarely the way you would expect by reading your manual.
When a member goes down nobody knows until it happens again — the array continues to serve as long as there are enough working members to do so. Weeks or months go by before someone reads the alarm.
Bringing a replacement disk on-line forces hours of continuous reads from each sector of each surviving disk. Ageing disks that have coasted for years die under those conditions; or one bad sector causes the entire rebuild to fail.
A disk drops offline months earlier, then it's forced back online. Its contents describe an older version of the volume than what is currently active and the controller merrily combines the two time periods into a single corrupted version of the current state.
All disks are operational but the controller that knows the disk order, stripe size, and parity rotation is either dead or has been replaced. All the data is still available — unlabelled — but can be fully recovered using virtual methods.
The rules that help keep a recoverable array recoverable.
A stopped rebuild has not lost anything yet. A pushed rebuild will grind failing heads across what may be the only copy of the data.
Order matters for stripe sets, but parity rotation makes the order of bays even more critical. While the order can be derived from the data, knowing the correct order of disks in advance could save many hours.
If a disk was removed weeks or months ago it likely had an older version of your data. If you try to force that disk back into service, you will write those older versions of your data over your current versions.
Often times, the better of the two failed disks is enough to allow you to recover everything. After imaging, you will have an array with only one disk missing; then parity takes care of restoring the remaining information.
The recovery of all data once one disk has been physically restored requires no further physical work. All other disks can be used to mathematically recover the missing information.
Each member of the RAID will be initially imaged read-only, regardless of its health. Once the failed pair has been assessed, the stronger of the two will be repaired – i.e., replacement heads, board work and/or firmware — until it can be imaged as well. In general, that single repair represents the entirety of the physical task to complete on the array. Since N–1 members of the array are now able to read data, each missing block becomes once again computable.
The geometry of the array – i.e., disk order, stripe size, parity placement/rotation and offset – is determined by analysing images taken of the disks. A virtual version of the array is then built using those images and outside the controllers. The file system is then rebuilt and validated from the newly created volume. RAID service provided in Manchester costs from £500 +VAT after a complimentary 48-hour diagnostic process and is subject to a 50% deposit if a member requires physical repairs. Please see what to do if your rebuild will not start and general recovery procedure for an array failure.
In most cases, yes. Parity exists for exactly one lost member; therefore, missing blocks can be computed from the remainder of the set. With two failures it becomes a physical job first: one of the failed disks is repaired until it can be imaged, and the maths handles the rest.
Generally, yes. Typically one will be in much worse shape than the other. When the stronger disk has been repaired and imaged, the array goes back to being a one-disk issue. If both members are badly damaged, there is still often value in attempting to recover some data from what reads.
Not if the surviving disks are either old, noisy, or already reporting errors. A rebuild reads every sector of every member for hours, exactly the type of workload that kills a second disk. First image the entire set. After imaging, if a rebuild fails it doesn't cost anything.
A rebuild requires reading every block of every surviving disk for an extended period. This type of heavy use can cause any relatively weak member to fail. Additionally, a single unreadable block on any member can prevent a rebuild from completing successfully on many controllers. And a rebuild is as hard a day's work for the survivors as they've ever had – at their age.
Each member of the array, including both bad members. Parity reconstruction requires N-1 disks with readable blocks. Usually, the best candidate for the last slot required is going to be the better of the two failed disks.
The cost is from £500 plus VAT once you have had your 48-hour free diagnostic. The prices are in the same bands as we use for all other RAID work. Where a member needs board-level repair, a 50% deposit will be required up front and payment of the remainder is not required until we can confirm that the recovery was successful.