Stalls at a percentage and stops
One of the remaining members can no longer read a specific sector. Without that sector the rebuild cannot calculate the missing data, so it cannot continue. Retrying hits the same sector.
Bad sectors
Rebuild operations can only fail when something in the array cannot be read. Running another rebuild will apply exactly the same amount of total I/O load on those disks as before, but will likely produce a worse outcome than the first run. If you have a parity-based RAID array such as RAID 5, this ends in one place: a second disk failure. RAID 5 data recovery provides complete coverage of the second disk failure.
A rebuild does not just read whatever it needs to in order to continue rebuilding; it also reads every single block of data from all of the remaining active disks (if they survive long enough). It writes as it goes, and on disks already carrying unreadable sectors that is the likeliest moment for a second failure and the point at which wrong data gets committed.
Once you understand how much of a load a rebuild places upon your disks, you should be able to identify each of the failure modes discussed below.
With parity based RAID arrays, the rebuild process calculates what was stored on the bad drive using calculations performed across every sector of every good drive. In other words, the rebuild is calculating what the bad drive had by performing its own version of "maths" across every good drive's data. Mirrors work slightly differently — during a rebuild, if a drive fails and has a mirrored copy available, then all of the good drives' data will simply be copied over into the new replacement drive. No matter whether you are working with a parity-based array or a mirror, the rebuild operation loads each individual disk as heavily as possible for an extended period of time (hours/days) as it attempts to build/recover the failed drive(s).
This makes sense because most of the time when the last drive fails, all the drives are the same age and purchased together. They have been exposed to the exact same usage for their entire lifespan since being installed. The rebuild will be a complete surface stress test on hardware that has the same cumulative amount of wear as the device that failed.
Determine which one you see — the approach will vary based upon what you see.
One of the remaining members can no longer read a specific sector. Without that sector the rebuild cannot calculate the missing data, so it cannot continue. Retrying hits the same sector.
The stress of rebuilding finished off a member that was already marginal. The array is now beyond its redundancy limits and therefore offline.
The array built to completion but the file system is broken — often due to the rebuild writing reconstructed data from stripes which were already inconsistent.
Typically a controller that can't get a consistent picture of what's in the set, or a member that drops out and then comes back periodically.
These are all the obvious things to do.
Do not run it again. The same read that you couldn't reach last time will fail again. The disks will have spent another few hours at full throttle trying to get there. If your marginally good member made it through the first pass, it may not survive the second.
Do not swap in more replacement disks. Adding another new disk will add even more to your writing and therefore the stress. The original data now sits behind two incomplete reconstructions.
Do not force a member online. Controllers offer to force-import or force-online disks they've rejected. That accepts a member whose contents the controller doesn't trust into an array they are about to write to.
You've now been told by the failed rebuild attempt that you can't fix the array where it sits. And that's useful.
This includes those which were ejected by the controller as well as those replacements that have been inserted.
What percentage did it reach, was another disk lost, and what did any errors say? This will help narrow down what could have gone wrong with the array before anything is opened.
All disks including the one that has failed and the replacement that was only partly written. Both will tell us what happened during the rebuild.
Each of your disks is copied (imaged), prior to any work being done on them. Therefore, a large array takes days before reconstruction even starts.
No. Rebuilds do not fail randomly. They fail because some of the members in the array can't be read and you'll get the same error applying the same max amount of I/O stress to those same failing disk(s) until the same failure occurs again. A marginally good member that survives one rebuild attempt may not survive another rebuild attempt.
A surviving member hit a bad sector it cannot read. Without that sector, the rebuild cannot compute the missing data. The percentage tells you approximately where the unreadable area lives. That's good to know.
Frequently. Rebuild stress caused a disk to fail. However, if the disk has simply been under too much pressure during a rebuild, the disk may still have a lot of life left. It just failed a prolonged read test, as opposed to failing dead. In many cases, such disks will image substantially after being treated gently.
That would be no. This option takes a member that your controller doesn't believe and puts it into an array you're about to write to. If the member's content is either out-of-date or only partially written, putting it online with force may allow that to spread through the entire set.
Your array built successfully; however, your file system appears to be damaged. Often the damage results from reconstructed data being computed from stripes that were already inconsistent. Thus this is typically a logical repair job versus an array repair issue.
All disks — every one identified by bay — including the failed original and any replacement that was partially written. Each carries information related to how things ended up, and the replacement occasionally holds usable reconstructed data.