Call us — 0161 871 0788
Mon–Fri · 9am–5:30pm · No fix, no fee
Start a free diagnostic →
← All case files // case file · NAS / RAID

The scrub was scheduled for Sunday.

A Synology reporting one failed disk and grinding audibly from a second bay, with a routine data scrub queued for the Sunday. Scrubbing is genuinely good practice. On that array in that state it would have finished it.

DeviceSynology · SHR over RAID 5
FaultOne member failed, one bearing
Turnaround12 days
Outcome97% recovered
MethodGraduated imaging · SHR unwind

What arrived

The unit reported a single failed disk and a degraded volume. What actually prompted the call was noise: a second bay grinding intermittently, loud enough to hear across the room. A replacement disk had already been ordered and the client intended to fit it at the weekend. They rang first. That phone call is the only reason this recovered as well as it did.

Why a scrub and a rebuild are the same dangerous operation

A Btrfs scrub on a Synology reads every allocated block on every member and verifies it against its stored checksum, repairing mismatches from parity or from the mirrored copy where one exists. It is genuinely valuable maintenance. Its whole purpose is to surface latent unreadable sectors before a rebuild needs them, because the worst possible moment to discover a bad sector is when it holds the only surviving copy of data being reconstructed.

Run on healthy hardware it is among the most useful things a NAS does. Run on an array that has already lost a member, it is a full-surface sequential read of every remaining disk, at queue depth, for as long as it takes — typically eight to thirty hours on consumer 4–8 TB disks — with no redundancy left to absorb a second failure.

The rebuild the client was planning is the identical workload. md reconstructs the replacement member by reading every stripe from every survivor and recomputing the missing chunk. Both operations touch everything, from everyone, without a safety net. On this array the disk under that continuous load would have been the one already grinding.

What the SMART data actually said

The interface reported one disk failed and the others healthy, which was true only in the sense that Synology’s threshold had not been crossed. Read directly, the picture was different.

The failed member had a head fault: attribute 5 (Reallocated_Sector_Ct) high and static, with attribute 187 (Reported_Uncorrect) climbing — the signature of a head that has stopped reading rather than a surface that is wearing out. The grinding member was the more interesting one. Its reallocated count was unremarkable, but attribute 197 (Current_Pending_Sector) was rising steadily and its spin-up time had drifted upward across the last several power cycles. That combination points at the fluid dynamic bearing rather than at the media, and it is not something a NAS surfaces, because no single attribute has tripped.

The two remaining members were genuinely serviceable but carried scattered pending sectors of their own — entirely ordinary for disks of that age, and precisely the latent damage a scrub exists to find. In a degraded array those sectors are unprotected: parity has already been spent covering the absent member, so a stripe with one bad sector on a survivor is a stripe with no route back.

Imaging order, and why it is decided before anything is plugged in

Nothing was mounted. All four disks were imaged read-only through a hardware write blocker, in an order chosen by condition rather than by bay number.

The bearing disk went first, because it was the one degrading with every minute of runtime. It was imaged in graduated passes: a fast sequential pass at a large block size with a short read timeout, collecting everything that came back without argument; then a reverse pass; then progressively smaller block sizes over the gaps only, with the drive rested between sessions to let the bearing cool. Roughly 99.4% came back before the noise changed character and it was stopped deliberately rather than run to destruction.

The failed member needed a donor head stack assembly, matched on drive family, head configuration and firmware revision — capacity alone is not a match, and a mismatched preamp will read nothing. With the donor fitted it imaged to about 96%, with the shortfall concentrated on the surfaces served by the original faulty head.

The two survivors imaged cleanly, their pending sectors resolving into a few hundred unreadable LBAs each.

SHR is not a RAID level

This is the part that catches generic recovery tools. Synology Hybrid RAID is not a layout in its own right — it is a scheme built on top of standard Linux components, and understanding the stack is what makes it recoverable.

SHR partitions each disk into regions sized to allow mixed capacities, builds a separate md array across matching regions, and then joins those arrays with LVM2 into a single logical volume carrying one Btrfs file system. A four-bay unit holding two different disk sizes is commonly two stacked md arrays under one volume group, not the single array the web interface implies.

So the reconstruction had layers, and each was solved from the disks rather than assumed:

The md layer. Superblock version 1.2 sits 4 KB from the start of each member partition and records the array UUID, the device role, the chunk size, the layout and an event count. Synology’s default chunk is 64 KB with left-symmetric parity rotation. Read from the images rather than trusted from the enclosure, those superblocks establish member order directly — and the event counters identify which member fell out first, which matters because a member that dropped days earlier holds stale data that must not be used to fill gaps.

The LVM layer. The LVM2 metadata area occupies the first megabyte of each physical volume and holds a plain-text descriptor of the volume group: which physical volumes belong to it, and how logical extents map onto physical ones. Parsed from the images, that established how the underlying arrays combined into the single volume the client saw.

The Btrfs layer. Btrfs keeps its primary superblock at 64 KB with mirrors at 64 MB and 256 GB, and being copy-on-write it never overwrites live metadata in place — a modified tree block is written to a new location and the root updated to point at it. That behaviour was decisive here. Where the current root referenced extents that fell in unreadable regions, superseded roots from earlier transactions still resolved, and rolling back to an older tree recovered directory structure that a journalling file system would have overwritten in place and lost.

Assembly and verification

Each md array was assembled virtually from the images, with the stale member excluded and its regions reconstructed from parity across the others instead. The volume group was mapped from the LVM metadata, the Btrfs file system mounted read-only from the assembled result, and the tree walked from the most recent root that resolved cleanly.

Files were verified by opening rather than by counting. Every extent that resolved into a region unreadable on both the failed and the bearing member simultaneously was recorded by path and reported as lost, rather than returned as a file that opens to garbage.

Outcome

About 97%. The shortfall was confined to stripes where the failed member’s bad surface and the bearing member’s pending sectors overlapped — the only regions on the array with no surviving copy anywhere.

Had either the scrub or the rebuild run first, the bearing disk would in all likelihood have seized under a continuous full-surface read. That takes the array two members down on single parity, which is not a degraded array but a destroyed one. The difference between 97% and almost nothing was a phone call made three days early.

// ready when you are

Facing something similar? Let's help.

Kick off with an instant online quote, or ring us and talk it through first. Either way you’ll know a clear, fixed price before any work starts.

Peter House, Oxford Street, Manchester M1 5AN · Mon–Fri 9am–5:30pm · No fix, no fee on most jobs