Bug 218353

This is likely a btrfs regression. Any filesystem that doesn't implement ->show_devname() will always show "/dev/root" in a scenario where a FS_REQUIRES_DEV filesytems is mounted (ext4, xfs, etc). btrfs uses a custom method to show the device name. I assume that a change in the v6.7-rc1 btrfs merge caused this regression. Report this to btrfs upstream via the mailing list if you care.

Again, whether something other than /dev/root is shown depends on the filesystem. You can't rely on this generically.

> This is likely a btrfs regression. Any filesystem that doesn't implement
> ->show_devname() will always show "/dev/root" in a scenario where a
> FS_REQUIRES_DEV filesytems is mounted (ext4, xfs, etc).

What I meant is when you boot directly without an initrd. Because then we always call:

create_dev("/dev/root", <device-number>);
mount_root_generic("/dev/root", ...);

which means the mnt->dev_name is set to "/dev/root" which the vfs directly retrieves and show you. Unless there's a specific ->show_devname() method. btrfs has that and does indeed show /dev/sd<whatever> before v6.7-rc1 and /dev/root after v6.7-rc1. While xfs continues to consistently show /dev/root.

> This is likely a btrfs regression. Any filesystem that doesn't implement
> ->show_devname() will always show "/dev/root" in a scenario where a
> FS_REQUIRES_DEV filesytems is mounted (ext4, xfs, etc).

What I meant is when you boot directly without an initrd. Because then we always call:

create_dev("/dev/root", <device-number>);
mount_root_generic("/dev/root", ...);

which means the mnt->dev_name is set to "/dev/root" which the vfs directly retrieves and show you. Unless there's a specific ->show_devname() method. btrfs has that and does indeed show /dev/sd<whatever> before v6.7-rc1 and /dev/root after v6.7-rc1. While xfs continues to consistently show /dev/root.

We have a ->show_devname, that code hasn't changed.  We did the mount v2 api change for 6.7, but I'm not entirely sure how that would have changed anything?  I booted with an initrd system and it's doing the correct thing, so it must be something in the non initrd boot path that is messing us up.

FWIW btrfs is just reporting back whatever we were passed in as the device name, so the fact that we were reporting /dev/sda* before and now /dev/root is likely a change outside of btrfs.  I'm having trouble figuring out how to boot without a initrd, so I can't reproduce yet to figure it out.

v6.7 doesn't have the mount api changes. I think that's coming with v6.8-rc1, maybe? On both v6.6 and v6.7 btrfs_mount_root()'s call to btrfs_scan_one_device() receives /dev/root as device from mount_block_root() yet /proc/self/mountinfo shows /dev/sda on v6.6 and /dev/root on v6.7.

I suspect that this is coming from the device_add_list() logic. FWIW, reverting the btrfs commits from v6.7-rc1 fixes the regression. So this definitely is from btrfs afaict. I suspect that the device scanning logic did cause ->latest_dev->name to be updated from /dev/root to /dev/sda and that somehow got broken in v6.7-rc1.

You're right, sorry I got confused which kernel was which.

The change that introduced it seems to be a5b8a5f9f835 ("btrfs: support cloned-device mount capability"), but I don't know for sure.

If that's the case then the answer is going to be find some other way to get the right device, especially given ext4 and xfs do the same thing.

This patch was to enable us to clone a device and mount multiple clones in different places without it freaking out about the UUID stuff, which means it doesn't get the device name change behavior anymore.  We'd prefer this behavior because this enables testing and also Valve's usecase for their Steam Deck.

I think it could be caused by bc27d6f0aa0e4d ("btrfs: scan but don't register device on single device filesystem"), by the logic the device name would not be remembered so it could change in the mountinfo. I have a initrd-less setup for VM testing (though not with btrfs as root), I'll try to bisect it.

Confirmed, 6.7 shows /dev/sda, reverting bc27d6f0aa0e4d shows /dev/root.

Correction, I must have been mistaken by mixing outputs of mount and proc files, 6.6 and 6.7 show /dev/root in /proc/self/mountinfo, I'm now testing 6.5 as it's in the original report.

root@localhost:~# cat /proc/self/mountinfo | grep sda
20 1 0:18 / / ro,relatime shared:1 - btrfs /dev/sda ro,discard=async,space_cache=v2,subvolid=5,subvol=/

root@localhost:~# uname -a
Linux localhost 99.6.0-gffc253263a13 #46 SMP PREEMPT_DYNAMIC Fri Jun  5 15:58:00 CEST 2015 x86_64 GNU/Linux

commit ffc253263a1375a65fa6c9f62a893e9767fbebfa
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Sun Oct 29 16:31:08 2023 -1000

    Linux 6.6

For me 6.5 still shows /dev/root so it can't reliably identify which commit caused that.

Thank you for the report.

The issue seems more complex than a simple scenario, as the following
test-case works well:

  $ mount /dev/sdb1 /btrfs
  $ cat /proc/self/mountinfo | grep btrfs
345 63 0:34 / /btrfs rw,relatime shared:179 - btrfs /dev/sdb1 rw,space_cache=v2,subvolid=5,subvol=/

However, the relevant part of the commit
bc27d6f0aa0e4de184b617aceeaf25818cc646de that may be failing could
be in identifying a device, whether it is the same or different
For this, we use:

     lookup_bdev(path, &path_devt);

and match with the devt(MAJ:MIN) saved in the btrfs_device;
would this work during initrd? I need to dig more. Trying
to figure out how can I reproduce this. Any idea?

Sorry - I'm one of the gentoo people bitten by this (https://bugs.gentoo.org/922809), the information in this thread seems inconsistent and to me this *is* a regression (something that worked for years doesn't work any more, and it clearly is the kernel since booting the same userspace with 6.6.12 works as ever).

(In reply to Christian Brauner from comment #3)
> > This is likely a btrfs regression. Any filesystem that doesn't implement
> > ->show_devname() will always show "/dev/root" in a scenario where a
> > FS_REQUIRES_DEV filesytems is mounted (ext4, xfs, etc).

> What I meant is when you boot directly without an initrd. Because then we
> always call:
> 
> create_dev("/dev/root", <device-number>);
> mount_root_generic("/dev/root", ...);
> 
> which means the mnt->dev_name is set to "/dev/root" which the vfs directly
> retrieves and show you. Unless there's a specific ->show_devname() method.
> btrfs has that and does indeed show /dev/sd<whatever> before v6.7-rc1 and
> /dev/root after v6.7-rc1. While xfs continues to consistently show /dev/root.

Exactly, in 6.7, /proc/self/mountinfo shows "btrfs /dev/root" and 6.6 "btrfs /dev/sda5". So the behavior changed towards, not away from the standard?

Regarding "create_dev("/dev/root", <device-number>);": As far as I understand it, this device would be created in the tmpfs that (always) provides early root, whether or not populated from an initramfs. After pivot_root() and mounting of devtmpfs on /dev, userspace would be responsible to populate it (udevadm trigger). Some information seems to be missing that would allow udevadm to create /dev/root, right? I'm not deep enough into these things to figure this out though.

Possible fix can be applied from here: https://github.com/kdave/btrfs-devel/commit/c9a96c234f18aeb5d5760a8e90a20971ec75be41

--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2304,6 +2304,7 @@ static int btrfs_unfreeze(struct super_block *sb)
 static int btrfs_show_devname(struct seq_file *m, struct dentry *root)
 {
        struct btrfs_fs_info *fs_info = btrfs_sb(root->d_sb);
+       char real[4096] = { "/dev/" };
 
        /*
         * There should be always a valid pointer in latest_dev, it may be stale
@@ -2311,7 +2312,8 @@ static int btrfs_show_devname(struct seq_file *m, struct dentry *root)
         * the end of RCU grace period.
         */
        rcu_read_lock();
-       seq_escape(m, btrfs_dev_name(fs_info->fs_devices->latest_dev), " \t\n\\");
+       scnprintf(real + 5, sizeof(real) - 5, "%pg", fs_info->fs_devices->latest_dev->bdev);
+       seq_puts(m, real);
        rcu_read_unlock();
 
        return 0;
---

I've verified that on my setup that it returns the real path of the device but would appreciate testing from the reported or whoever is affected. Thanks.

(In reply to David Sterba from comment #14)
> Possible fix can be applied from here:
> https://github.com/kdave/btrfs-devel/commit/
> c9a96c234f18aeb5d5760a8e90a20971ec75be41

Thanks David!
That patch applied cleanly to 6.7.3-gentoo sources but did not make the required /dev/root alias appear. Therefore grub-mkconfig / update-grub still fail.

(In reply to Bernd Feige from comment #15)
> (In reply to David Sterba from comment #14)
> > Possible fix can be applied from here:
> > https://github.com/kdave/btrfs-devel/commit/
> > c9a96c234f18aeb5d5760a8e90a20971ec75be41
> 
> Thanks David!
> That patch applied cleanly to 6.7.3-gentoo sources but did not make the
> required /dev/root alias appear. Therefore grub-mkconfig / update-grub still
> fail.

Sorry - update: with this patch, /proc/self/mountinfo shows the real device (/dev/sda5 in my case) again, but the first run of grub-mkconfig failed anyway. Running once with a /dev/root symlink made it succeed.

But the next grub-mkconfig run (without reboot, just removed my /dev/root symlink again) succeeded without that symlink, so the patch will probably fix the behavior as it reverts the reported device to be the real (and existing) device.

(In reply to David Sterba from comment #14)
> Possible fix can be applied from here:
> https://github.com/kdave/btrfs-devel/commit/
> c9a96c234f18aeb5d5760a8e90a20971ec75be41

I tested the patch on Linux 6.8-rc1 and "/proc/self/mountinfo" successfully showed "/dev/sda8" instead of "/dev/root"

But then when I did "update-grub" without using the "ln -s /dev/sda8 /dev/root" workaround, I got the "/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?)." error

Not sure why "update-grub" is not working even though "/proc/self/mountinfo" shows "/dev/sda8" instead of "/dev/root" when using the patch

(In reply to CHECK_1234543212345 from comment #17)
> (In reply to David Sterba from comment #14)
> > Possible fix can be applied from here:
> > https://github.com/kdave/btrfs-devel/commit/
> > c9a96c234f18aeb5d5760a8e90a20971ec75be41
> 
> I tested the patch on Linux 6.8-rc1 and "/proc/self/mountinfo" successfully
> showed "/dev/sda8" instead of "/dev/root"
> 
> But then when I did "update-grub" without using the "ln -s /dev/sda8
> /dev/root" workaround, I got the "/usr/sbin/grub-probe: error: cannot find a
> device for / (is /dev mounted?)." error
> 
> Not sure why "update-grub" is not working even though "/proc/self/mountinfo"
> shows "/dev/sda8" instead of "/dev/root" when using the patch

After rebooting, I also see the same thing - "update-grub" or grub-mkconfig does not work even though /proc/self/mountinfo shows the right device now.
Interestingly, the observation above (works without /dev/root symlink after running it once with a /dev/root symlink in place) remains. But after reboot it still doesn't.

Could you please provide the kernel logs from boot, including the time when 'update-grub' fails?

Is there any documentation on how to set up a test environment to reproduce the problem locally? I have tried with Fedora, Arch Linux, and Oracle Linux (and remove initrd), but have had no luck so far.

(In reply to Anand Jain from comment #19)
> Could you please provide the kernel logs from boot, including the time when
> 'update-grub' fails?

I uploaded it here:
https://www.dropbox.com/scl/fi/b3q7ck17mliue51p510l4/boot.log?rlkey=nh3inv6k58k900mt93o2rgjfo&dl=0
you can see the times when grub was run by the os-prober logs, as I also activated os-prober.

> Is there any documentation on how to set up a test environment to reproduce
> the problem locally? I have tried with Fedora, Arch Linux, and Oracle Linux
> (and remove initrd), but have had no luck so far.

I can describe the procedure but don't want to bore anybody.
First thing: The distribution used should not matter. All distributions boot through initrd. I don't know what you mean by "remove initrd" - it involves compiling a kernel with all drivers compiled in (not as module) which are necessary to mount the btrfs root, including bus, sata / MTD and FS. Then you point grub to that kernel to load it.

According to the kernel messages, btrfs.ko updated the path /dev/root to /dev/sda5.


   Feb 04 12:00:56 vivo kernel: BTRFS info: devid 1 device path /dev/root changed to /dev/sda5 scanned by mount (2828)


Are we sure this is a regression from the commit bc27d6f0aa0e ("btrfs: scan but don't register device on single device filesystem"); I'm not exactly sure how?.

Appreciate the steps; my tests match as well, but the mountinfo still shows the actual device. What else could it be..

I'm puzzled why update-grub still fails, does it not use the path in /proc/self/mountinfo at all? Meanwhile I found a bit different fix not using the temporary buffer the but the end result would be the same, ie the /dev/sdx path.

I'm suspecting that there's a cached value somewhere that gets used instead of output of the mountinfo. If this is grub1 we're talking about, there's a file /boot/grub/device.map that maps config raw values to real devices:

(hd0)   /dev/disk/by-id/ata-ST3160815AS_6RA09W98

Could it happen that once the first wrong /dev/root appeared in the mount info it got saved to this file? Then any time you try to run update-grub it would look up /dev/root.

I'm not aware of similar mapping file for grub2 but there could be a similar problem.

It's somewhere in the device probing and/or consistency of the mountinfo entries.

Grub-probe knows about /dev/root and should be able to look up the real device:

https://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/osdep/unix/getroot.c#n493

although it first parses mountinfo (on linux):

https://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/osdep/linux/getroot.c#n378

For context (rom the original report):

8 1 0:16 / / rw,noatime - btrfs /dev/sda8 ... (working)

vs

17 1 0:15 / / rw,noatime - btrfs /dev/root ... (not working)

The 0:16 and 0:15 are device major:minor numbers and this is what grub-probe reads:

https://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/osdep/linux/getroot.c#n413

Later it also reads the fstype and device path, and if everything worked until now tries to read devices grub_find_root_devices_from_btrfs().

My suspicion was that major:minor must match the device node, thus if we for some reason have the numbers corresponding to /dev/root but still return the /dev/sdx real device then it fails. Or vice versa (sdx major:minor vs /dev/root). But seems that there's no logic like that.

So the major:minor are some internal identifiers of mount, not corresponding to the block devices and not relevant here.

Good news, I have a fix (and actually understand the whole problem).

Please test https://github.com/kdave/btrfs-devel/commit/b80f3ec6592c69f88ebc74a4e16676af161e2759 thanks.

Note that device scan is necessary after the root fileystem is mounted, this seems to be done on systems with proper userspace boot sequence (ie. there's the line about name change in the system log).

My system boots almost directly to shell where the mount is done by the swapper task:

BTRFS: device fsid 2b80ef5f-9d2d-41b5-8a08-9dbdf2f3f065 devid 1 transid 114 /dev/root scanned by swapper/0 (1)

In that state the following does not work (yet):

$ grub2-probe /
grub2-probe: error: cannot find a device for / (is /dev mounted?).

Then

$ btrfs device scan
BTRFS info: devid 1 device path /dev/root changed to /dev/sda scanned by btrfs (130)
$ grub2-probe /
...
btrfs

(In reply to David Sterba from comment #27)
> Please test
> https://github.com/kdave/btrfs-devel/commit/
> b80f3ec6592c69f88ebc74a4e16676af161e2759 thanks.

I tested the patch on Linux 6.8-rc1 and "/proc/self/mountinfo" successfully showed "/dev/sda8" instead of "/dev/root"

And when I did "update-grub" without using the "ln -s /dev/sda8 /dev/root" workaround, "update-grub" successfully worked for me!!!

Thanks for successfully fixing "update-grub" not working when booting Linux without initrd for me!!!

(In reply to David Sterba from comment #27)
> Please test
> https://github.com/kdave/btrfs-devel/commit/
> b80f3ec6592c69f88ebc74a4e16676af161e2759 thanks.

Thanks Dave! That fixes the problems (of grub, and of udisksd complaining "Error statting /dev/root: No such file or directory") for me as well - tested against linux-6.7.3-gentoo.

Thanks for the testing! Let me know if want the Reported-by and Tested-by tags in the commit (not everybody is fine with that so I'm asking, https://docs.kernel.org/process/5.Posting.html?highlight=reported-by#patch-formatting-and-changelogs).

(In reply to David Sterba from comment #31)
> Thanks for the testing! Let me know if want the Reported-by and Tested-by
> tags in the commit (not everybody is fine with that so I'm asking,
> https://docs.kernel.org/process/5.Posting.html?highlight=reported-by#patch-
> formatting-and-changelogs).

Fine with me.

(In reply to David Sterba from comment #31)
> Thanks for the testing! Let me know if want the Reported-by and Tested-by
> tags in the commit (not everybody is fine with that so I'm asking,
> https://docs.kernel.org/process/5.Posting.html?highlight=reported-by#patch-
> formatting-and-changelogs).

Yep, it is fine with me too

New fix:

 https://patchwork.kernel.org/project/linux-btrfs/patch/8dd1990114aabb775d4631969f1beabeadaac5b7.1707132247.git.anand.jain@oracle.com/

(In reply to Anand Jain from comment #34)
> New fix:
> 
>  https://patchwork.kernel.org/project/linux-btrfs/patch/
> 8dd1990114aabb775d4631969f1beabeadaac5b7.1707132247.git.anand.jain@oracle.
> com/

I tested that patch and I confirm that "update-grub" successfully worked for me with that patch when booting Linux without initrd!

What happened to the fix for this? Was it merged? It looks like it was dropped, but I got the feeling I'm missing something here.

Another bug has been reported, so now we need to address both issues. Thus, it's back to the drawing board for the fixes.

RFC v3 with the fix has been sent. Please verify. Also, provide the device maj:min (ls -li) for /dev/root and the actual root device /dev/sdx. Thx

https://lore.kernel.org/linux-btrfs/e2add8d54fbbd813305ba014c11d21d297ad87d0.1709782041.git.anand.jain@oracle.com/T/#u

(In reply to Anand Jain from comment #38)
> RFC v3 with the fix has been sent. Please verify. Also, provide the device
> maj:min (ls -li) for /dev/root and the actual root device /dev/sdx. Thx
> 
> https://lore.kernel.org/linux-btrfs/e2add8d54fbbd813305ba014c11d21d297ad87d0.
> 1709782041.git.anand.jain@oracle.com/T/#u

I tested the new "RFC v3" patch and I confirm that "update-grub" successfully worked for me when booting Linux without initrd without having to do the "ln -s /dev/sda8 /dev/root" workaround!

"ls -li /dev/sda8" shows:
132 brw-rw---- 1 root disk 8, 8 Mar  7 17:41 /dev/sda8

Thanks for verifying v3. There was also a v4 RFC, which is a rebase on the mainline kernel.

https://lore.kernel.org/linux-btrfs/CAKLYge+crQ73TJMvd6k2YtLSnT8rQ44AYb5q1HCkL4GB+gb4Xg@mail.gmail.com/T/#t

Plus, it obtains the devt from the bdev instead of lookup_bdev().

If v4 can be verified, that would be great. However, if v3 has worked, looking at the changes from v3 to v4, v4 will work as well.

Also, how about ls -li for both /dev/root and /dev/sda8 (root device)? Thx.

(In reply to Anand Jain from comment #40)
> Thanks for verifying v3. There was also a v4 RFC, which is a rebase on the
> mainline kernel.
> 
> https://lore.kernel.org/linux-btrfs/
> CAKLYge+crQ73TJMvd6k2YtLSnT8rQ44AYb5q1HCkL4GB+gb4Xg@mail.gmail.com/T/#t
> 
> Plus, it obtains the devt from the bdev instead of lookup_bdev().
> 
> If v4 can be verified, that would be great. However, if v3 has worked,
> looking at the changes from v3 to v4, v4 will work as well.
> 
> Also, how about ls -li for both /dev/root and /dev/sda8 (root device)? Thx.

I tested the new "RFC v4" patch and I confirm that "update-grub" successfully worked for me when booting Linux without initrd without having to do the "ln -s /dev/sda8 /dev/root" workaround!

"ls -li /dev/sda8" shows:
132 brw-rw---- 1 root disk 8, 8 Mar  7 19:24 /dev/sda8

"ls -li /dev/root" shows:
ls: cannot access '/dev/root': No such file or directory

(In reply to Anand Jain from comment #40)
> Thanks for verifying v3. There was also a v4 RFC, which is a rebase on the
> mainline kernel.
> 
> https://lore.kernel.org/linux-btrfs/
> CAKLYge+crQ73TJMvd6k2YtLSnT8rQ44AYb5q1HCkL4GB+gb4Xg@mail.gmail.com/T/#t
> 
> Plus, it obtains the devt from the bdev instead of lookup_bdev().
> 
> If v4 can be verified, that would be great. However, if v3 has worked,
> looking at the changes from v3 to v4, v4 will work as well.
> 
> Also, how about ls -li for both /dev/root and /dev/sda8 (root device)? Thx.

Tested v4 RFC now as well, works for me!

LANG=C /bin/ls -li /dev/root /dev/sda5
/bin/ls: cannot access '/dev/root': No such file or directory
106 brw-rw---- 1 root disk 8, 5 Mar  9 11:45 /dev/sda5

Thanks!

So, any update/idea as to which kernel version this will be included in?
(currently stuck at 6.6.x due to this..)

The fix d565fffa68560ac540bf3d62cc79719da50d5e7a is now in Linus' tree and will be in 6.9-rc1. The stable trees need a minor update which mathes the v4 of the patch https://lore.kernel.org/linux-btrfs/65a11e853a31b18b620f31cbbddf03e277fe3edf.1709809171.git.anand.jain@oracle.com/

Expected ETA for stable release is 6.8.3 and 6.7.12, i.e. +0.0.2 from now due to the timings and stable workflow pipeline.

Thanks everybody, the final fix took longer than expected, sorry for that.

Fix released in 6.7.12.

And 6.8.3.