Bug 210501

Summary:	btrfs replace checks source device size instead of source filesystem size
Product:	File System	Reporter:	Andrej Podzimek (andrej)
Component:	btrfs	Assignee:	BTRFS virtual assignee (fs_btrfs)
Status:	NEW ---
Severity:	normal	CC:	nazar
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	5.9.11	Subsystem:
Regression:	No	Bisected commit-id:

Description Andrej Podzimek 2020-12-05 18:26:27 UTC

I was replacing a 5TB drive by a 4TB one (in a raid6 array with raid1c3 metadata). Before the replacement I did the following:

  # btrfs filesystem resize 5:-2T /raid

I thought this^^^ would make the replacement possible, but it didn't:

  # btrfs replace start -r /dev/mapper/crypt5 /dev/mapper/crypt5_new /raid
  ERROR: target device smaller than source device (required 5000978980864 bytes)

What eventually helped was this:

  # cryptsetup resize --size=$(((5000978980864 - 1024**4 * 3 / 2) / 512)) crypt5

(I subtracted 1.5 TB (instead of 2 TB) from the LUKS container size to get below target device size without harming Btrfs data.)

After the LUKS size reduction I could initiate the replacement without issues.

I think that Btrfs should check "source filesystem size" against "destination device size". It appears to be using "source device size" in the comparison.

Comment 1 Andrej Podzimek 2023-02-20 13:04:53 UTC

A workaround for future readers: https://unix.stackexchange.com/a/513352/353114

The TL;DR is that you need to:

0. downsize the Btrfs filesystem on the source to <= the destination’s size.
1. use the source device’s *index*, not its name/path, in the replace command.

When you use the index, the btrfs tool will know that the FS has been downsized to a size smaller than the device. If you use the device path, the full size of the device will be considered and compared, despite the fact that Btrfs is no longer using the full size.

The workaround I used above 2+ years ago worked fine, but was too complicated. There was no need to (also) resize the LUKS container. What was needed instead:

  # btrfs filesystem resize 5:-2T /raid
  # btrfs replace start -r 5 /dev/mapper/crypt5_new /raid

Last but not least, the "-2T" was too sloppy and required an additional enlargement of the FS after the device swap. It would have been easier to downsize the FS on the source device directly to the exact target device size before the swap.

Comment 2 Nazar Mokrynskyi 2024-01-14 17:49:34 UTC

I just hit this again and I wish I knew the workaround sooner, device removal is so painfully slow on btrfs. For context, I am replacing one raw disk with LUKS on top of the disk of identical size, so the resulting size there is tiny bit smaller than raw disk.