A Netgear ReadyNAS reporting bad sectors across several members and running badly. The client added a disk expecting the unit to rebuild and stabilise itself. The expansion started, and the array never came back.
← All case files · from £500 + VAT
The NAS had been unwell for a while — bad sectors reported on more than one member, sluggish performance, intermittent access. Not a dramatic failure but a quiet degradation, the sort that gets lived with.
The client added a disk, reasoning that more capacity and a rebuild would let the system sort itself out. The array failed outright during the expansion, the unit stopped booting, and the business lost access to documents, databases and project files.
Every surviving disk is read end to end and the missing member recomputed from parity. Heavy sustained work, and the reason a second disk so often fails during one — the rebuild is the first time in years that every sector on every disk has been asked for.
Adding a member changes the stripe geometry itself. A four-disk RAID 5 stripes across three data chunks plus parity; add a fifth and it stripes across four plus parity. Every block therefore belongs at a different physical offset than it did before, so the whole array is read and written back in the new arrangement.
Maximum sustained read and write load on every member simultaneously, for hours or days depending on capacity. A reshape of a multi-terabyte array routinely runs longer than a rebuild of the same array, because it is moving everything rather than reconstructing one share of it.
The reshape reaches a region it cannot read, stalls or aborts partway through, and leaves the array in neither the old geometry nor the new one. Which is precisely why the unit would not assemble afterwards.
This is the technical heart of the job and it is worth being exact about, because it is what separates a stalled expansion from an ordinary failed array.
A Linux md reshape works forward through the array and records how far it has got. The 1.x superblock on each member carries a reshape position: everything below that offset has been rewritten into the new geometry, everything above it is still in the old one. Because the first stripes have nowhere safe to be written while they are also being read, the critical opening section is copied out to a backup file first, and the superblock records where that lives.
Interrupt the process and you inherit all of that. To read the volume you must know, for any given logical address, whether it currently lives under the old layout or the new one — and the answer depends entirely on which side of the reshape position it falls. Get that boundary wrong by a single stripe and everything above it decodes to noise.
Normally md resumes from the recorded position and finishes the job. That was not available here, for two reasons. The reshape had aborted on unreadable sectors, so resuming would immediately meet the same sectors again with even less redundancy in hand. And the backup file lived on a volume that was itself part of the array.
Read directly rather than through the enclosure, the members told a consistent story. Two carried elevated attribute 5 (Reallocated_Sector_Ct) with attribute 197 (Current_Pending_Sector) climbing steadily — sectors the drive had tried to read, failed on, and marked for reallocation on the next write that never came. A third had a smaller but growing pending count.
None had crossed the threshold that makes a NAS declare a member failed, which is exactly the trap. The unit reported an array that was degraded and serving data, and a user reasonably concluded that adding capacity was a repair.
Worth noting the underlying probability, because it explains why expansions on marginal hardware fail so reliably. A reshape reads every sector on every member. If a drive carries even a handful of unreadable sectors, the chance the reshape encounters one is not small — it is certain, because the reshape reads all of them by definition.
Every member was imaged first, including the partially written new disk, which mattered because it held the only copy of whatever had already been redistributed into the wider geometry.
The three degraded members were imaged in graduated passes: a fast sequential sweep at a large block size with a short timeout to bank everything readable, then repeated passes over the gaps with longer timeouts and rest intervals between sessions. Roughly 99.97% of each came back that way, with the residue concentrated in the reallocation zones.
Reassembly then had to solve the boundary problem. The reshape position was read from the member superblocks and cross-checked against the images themselves — the transition between geometries is visible in the data, because file system structures that resolve coherently under one arrangement resolve to nothing under the other. Both layouts were reconstructed independently, and the volume assembled by applying the old geometry above the boundary and the new geometry below it, verified region by region against whether the file system made sense rather than assumed from the metadata alone.
The critical-section backup was located in the images and used to restore the opening stripes the reshape had displaced.
About 99%, with the shortfall confined to regions unreadable across enough members simultaneously that parity could not fill them — the same reallocation zones that had stalled the reshape in the first place.
The data went back on fresh disks in a new enclosure. The original members were not reused.
The bad sectors alone would not have caused this. The array was degrading and still serving data, and would have kept doing so long enough to copy everything off at leisure. What made it unrecoverable by ordinary means was the expansion — an operation begun with entirely good intentions on hardware that could not survive it.
Adding capacity does not treat failing disks. Bad sectors are a hardware condition; extra storage does not repair them, does not retire the failing members and does not reduce load. It applies the heaviest available load to the components least able to take it. A degraded array is not short of space. It is short of healthy disks.
The correct order is unexciting. Copy the data off while it still reads. Then replace the failing disks one at a time, letting each resync complete before touching the next. Expansion comes afterwards, on hardware that is well. NAS and RAID recovery is from £500 +VAT after a free 48-hour diagnostic.
Drop the drive at our Oxford Street reception, or post it to us — it costs nothing to find out what happened. You get a written figure from the fixed bands before any work begins.