Bug 210521

Summary:	btrfs replace doesn't honor the -r option on raid5 and raid6
Product:	File System	Reporter:	Andrej Podzimek (andrej)
Component:	btrfs	Assignee:	BTRFS virtual assignee (fs_btrfs)
Status:	NEW ---
Severity:	low
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	5.9.11	Subsystem:
Regression:	No	Bisected commit-id:

Description Andrej Podzimek 2020-12-06 19:27:42 UTC

I gradually replaced all disks in an array (-d raid6 -m raid1c3) with SSDs.

For the last replacement I was expecting a lightning fast progress from the -r option, because all other volumes were already SSDs and I thought that -r would avoid reading from the last spinning disk and restore its data from parity instead.

This was not the case and the data transfer rate during the last device replacement was limited by the last spinning disk's capabilities. The spinning disk was, indeed, heavily utilized (based on iostat).

This could become a problem if (e.g.) the spinning disk was faulty and reading at 1 MB/s. In that case one may have to remove the slow/failing disk first and then specify the device to replace using a devid. Yet that comes extra risk (e.g. pulling the wrong drive) and is hard to carry out remotely on a machine with available hot spares.

(As a side note, the btrfs-replace man page only mentions mirrors in its description of -r; it doesn't say whether and how -r works with raid5/6.)

Comment 1 Andrej Podzimek 2021-02-24 18:05:45 UTC

In retrospect, I guess this is/was because the parity is not checksummed, so parity blocks have to be recalculated from the other drives to make sure everything is correct. That's fine.

But in that case I would expect, for an N-drive array, a load of (100% / (N - redundancy)) on drives other than the replaced one. This was quite obviously not the case; the load on the other drives was like >50% on a 8-drive array.

(Admittedly, I can't rule out the possibility that the relativey high load was in fact a symptom of a *low* load combined with power saving features and inefficiency that comes with them.)