Created attachment 132661 [details] orignal 3.13.x config After trying to move a 3.13.x stable based system to 3.14.x, the system which uses a cciss based array for storage is unable to boot. The problem was bisected to the following commit: commit 63541927c8d11d2686778b1e8ec71c14b4fd53e4 Btrfs: add support for inode properties After the bisect was tried a config based on the original one, with the exception that BTRFS support was DISABLED. The system booted normally, even though the kernel version was v3.15-rc1
I used these command during the git bisect run: cp ../config-v3.13-debug ./.config; make olddefconfig; make olddefconfig; make cp ./.config ../1/config-`git describe`-debug; cp arch/x86/boot/bzImage ../1/vmlinuz-`git describe`-debug; make mrproper
git bisect run details: git bisect log git bisect start # bad: [c9eaa447e77efe77b7fa4c953bd62de8297fd6c5] Linux 3.15-rc1 git bisect bad c9eaa447e77efe77b7fa4c953bd62de8297fd6c5 # good: [f994ec5a5d8094a195c23b08d373b055ef0044aa] Linux 3.13.10 git bisect good f994ec5a5d8094a195c23b08d373b055ef0044aa # good: [d8ec26d7f8287f5788a494f56e8814210f0e64be] Linux 3.13 git bisect good d8ec26d7f8287f5788a494f56e8814210f0e64be # bad: [d265d9ac6c7c3201f0fea737cdf9c74e50415178] [media] exynos4-is: Use external s5k6a3 sensor driver git bisect bad d265d9ac6c7c3201f0fea737cdf9c74e50415178 # good: [82c477669a4665eb4e52030792051e0559ee2a36] Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 82c477669a4665eb4e52030792051e0559ee2a36 # good: [ca2a650f3dfdc30d71d21bcbb04d2d057779f3f9] Merge branch 'for-linus' of git://git.infradead.org/users/vkoul/slave-dma git bisect good ca2a650f3dfdc30d71d21bcbb04d2d057779f3f9 # bad: [0f44bc36ba1cddbd34371110c4ef8f0800c8d465] Merge branch 'for-linus' of git://git.samba.org/sfrench/cifs-2.6 git bisect bad 0f44bc36ba1cddbd34371110c4ef8f0800c8d465 # good: [cdfc83075fb76369a31e6c187d0cebcab9f8b9c8] Merge branch 'upstream' of git://git.linux- mips.org/pub/scm/ralf/upstream-linus git bisect good cdfc83075fb76369a31e6c187d0cebcab9f8b9c8 # good: [6c3df5da67f1f53df78c7e20cd53a481dc28eade] [media] media: v4l2-dev: fix video device index assignment git bisect good 6c3df5da67f1f53df78c7e20cd53a481dc28eade # bad: [e30b82bbe098d9514ed0e9b5ec372daf7429e0f7] Merge tag 'jfs-3.14' of git://github.com/kleikamp/linux-shaggy git bisect bad e30b82bbe098d9514ed0e9b5ec372daf7429e0f7 # bad: [580f0a678ebeba85d30b6a7f22ce32c472263c72] Btrfs: fix extent_from_logical to deal with skinny metadata git bisect bad 580f0a678ebeba85d30b6a7f22ce32c472263c72 # good: [2ef1fed285d58e77ce777d9a525fed4788b5a6d0] Btrfs: more efficient push_leaf_right git bisect good 2ef1fed285d58e77ce777d9a525fed4788b5a6d0 # good: [c9ea7b24ce5863d65efb1134319cede160674d41] Btrfs: stop caching thread if extent_commit_sem is contended git bisect good c9ea7b24ce5863d65efb1134319cede160674d41 # good: [2c9ee85671f66cd3ffc7067de47cc59ed6677299] btrfs: Add noflushoncommit mount option. git bisect good 2c9ee85671f66cd3ffc7067de47cc59ed6677299 # good: [8e56338d7d0ee38ecae86d35dae43020356acca1] Btrfs: remove unnecessary transaction commit before send git bisect good 8e56338d7d0ee38ecae86d35dae43020356acca1 # bad: [63541927c8d11d2686778b1e8ec71c14b4fd53e4] Btrfs: add support for inode properties git bisect bad 63541927c8d11d2686778b1e8ec71c14b4fd53e4 # good: [1acae57b161ef1282f565ef907f72aeed0eb71d9] Btrfs: faster file extent item replace operations git bisect good 1acae57b161ef1282f565ef907f72aeed0eb71d9 63541927c8d11d2686778b1e8ec71c14b4fd53e4 is the first bad commit commit 63541927c8d11d2686778b1e8ec71c14b4fd53e4 Author: Filipe David Borba Manana <fdmanana@gmail.com> Date: Tue Jan 7 11:47:46 2014 +0000 Btrfs: add support for inode properties This change adds infrastructure to allow for generic properties for inodes. Properties are name/value pairs that can be associated with inodes for different purposes. They are stored as xattrs with the prefix "btrfs." Properties can be inherited - this means when a directory inode has inheritable properties set, these are added to new inodes created under that directory. Further, subvolumes can also have properties associated with them, and they can be inherited from their parent subvolume. Naturally, directory properties have priority over subvolume properties (in practice a subvolume property is just a regular property associated with the root inode, objectid 256, of the subvolume's fs tree). This change also adds one specific property implementation, named "compression", whose values can be "lzo" or "zlib" and it's an inheritable property. The corresponding changes to btrfs-progs were also implemented. A patch with xfstests for this feature will follow once there's agreement on this change/feature. Further, the script at the bottom of this commit message was used to do some benchmarks to measure any performance penalties of this feature. Basically the tests correspond to: Test 1 - create a filesystem and mount it with compress-force=lzo, then sequentially create N files of 64Kb each, measure how long it took to create the files, unmount the filesystem, mount the filesystem and perform an 'ls -lha' against the test directory holding the N files, and report the time the command took. Test 2 - create a filesystem and don't use any compression option when mounting it - instead set the compression property of the subvolume's root to 'lzo'. Then create N files of 64Kb, and report the time it took. The unmount the filesystem, mount it again and perform an 'ls -lha' like in the former test. This means every single file ends up with a property (xattr) associated to it. Test 3 - same as test 2, but uses 4 properties - 3 are duplicates of the compression property, have no real effect other than adding more work when inheriting properties and taking more btree leaf space. Test 4 - same as test 3 but with 10 properties per file. Results (in seconds, and averages of 5 runs each), for different N numbers of files follow. * Without properties (test 1) file creation time ls -lha time 10 000 files 3.49 0.76 100 000 files 47.19 8.37 1 000 000 files 518.51 107.06 * With 1 property (compression property set to lzo - test 2) file creation time ls -lha time 10 000 files 3.63 0.93 100 000 files 48.56 9.74 1 000 000 files 537.72 125.11 * With 4 properties (test 3) file creation time ls -lha time 10 000 files 3.94 1.20 100 000 files 52.14 11.48 1 000 000 files 572.70 142.13 * With 10 properties (test 4) file creation time ls -lha time 10 000 files 4.61 1.35 100 000 files 58.86 13.83 1 000 000 files 656.01 177.61 The increased latencies with properties are essencialy because of: *) When creating an inode, we now synchronously write 1 more item (an xattr item) for each property inherited from the parent dir (or subvolume). This could be done in an asynchronous way such as we do for dir intex items (delayed-inode.c), which could help reduce the file creation latency; *) With properties, we now have larger fs trees. For this particular test each xattr item uses 75 bytes of leaf space in the fs tree. This could be less by using a new item for xattr items, instead of the current btrfs_dir_item, since we could cut the 'location' and 'type' fields (saving 18 bytes) and maybe 'transid' too (saving a total of 26 bytes per xattr item) from the btrfs_dir_item type. Also tried batching the xattr insertions (ignoring proper hash collision handling, since it didn't exist) when creating files that inherit properties from their parent inode/subvolume, but the end results were (surprisingly) essentially the same. Test script: $ cat test.pl #!/usr/bin/perl -w use strict; use Time::HiRes qw(time); use constant NUM_FILES => 10_000; use constant FILE_SIZES => (64 * 1024); use constant DEV => '/dev/sdb4'; use constant MNT_POINT => '/home/fdmanana/btrfs-tests/dev'; use constant TEST_DIR => (MNT_POINT . '/testdir'); system("mkfs.btrfs", "-l", "16384", "-f", DEV) == 0 or die "mkfs.btrfs failed!"; # following line for testing without properties #system("mount", "-o", "compress-force=lzo", DEV, MNT_POINT) == 0 or die "mount failed!"; # following 2 lines for testing with properties system("mount", DEV, MNT_POINT) == 0 or die "mount failed!"; system("btrfs", "prop", "set", MNT_POINT, "compression", "lzo") == 0 or die "set prop failed!"; system("mkdir", TEST_DIR) == 0 or die "mkdir failed!"; my ($t1, $t2); $t1 = time(); for (my $i = 1; $i <= NUM_FILES; $i++) { my $p = TEST_DIR . '/file_' . $i; open(my $f, '>', $p) or die "Error opening file!"; $f->autoflush(1); for (my $j = 0; $j < FILE_SIZES; $j += 4096) { print $f ('A' x 4096) or die "Error writing to file!"; } close($f); } $t2 = time(); print "Time to create " . NUM_FILES . ": " . ($t2 - $t1) . " seconds.\n"; system("umount", DEV) == 0 or die "umount failed!"; system("mount", DEV, MNT_POINT) == 0 or die "mount failed!"; $t1 = time(); system("bash -c 'ls -lha " . TEST_DIR . " > /dev/null'") == 0 or die "ls failed!"; $t2 = time(); print "Time to ls -lha all files: " . ($t2 - $t1) . " seconds.\n"; system("umount", DEV) == 0 or die "umount failed!"; Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com> Signed-off-by: Josef Bacik <jbacik@fb.com> Signed-off-by: Chris Mason <clm@fb.com> :040000 040000 9e488f01f6d7b0723eaad9987f53079f3440f80d 88681950b323c2108c7367ac655f8cd0a05a10c6 M fs :040000 040000 71b5fc062559d305452666d016c061294fcee8ea 4cd01b47dc7e4f6c72aec55cfba8b338a30e2025 M include
Created attachment 132671 [details] screen capture of BTRFS-enabled kernel opps WITH bisected commit included Here is a screen capture of the BTRFS-enabled kernel opps, on a kernel WITH the bisected commit included
I use the kernel built with the attached config on one physical and a couple of virtual machines. The problems described so far were tested only on the physical machine - me thinking that the virtual ones would be somehow unaffected. Today I tested 3.14.1 on one of the VMs - same oops. This in light of my worries that somehow my bisection had gone wrong - but testing proves otherwise. Is there something I missed that needs to be done when migrating systems with BTRFS root from older kernels to 3.14? The physical machine's root is now switched from BTRFS to XFS, and was left running 3.15-rc1 but without BTRFS support in the kernel.
Created attachment 133111 [details] here is how 3.14.1 panics This is what the original panic that prompted this bisect looks like - this is 3.14.1 compiled with my original config, after make olddefconfig, trying to boot a VM.
So, I made the following patch (available as an attached file also): --- a/init/do_mounts.c +++ b/init/do_mounts.c @@ -359,6 +359,8 @@ static void __init get_fs_names(char *page) static int __init do_mount_root(char *name, char *fs, int flags, void *data) { struct super_block *s; + printk(KERN_INFO + "--> do_mount_root: name = %s, fs = %s, flags = %d.\n", name, fs, flags); int err = sys_mount(name, "/root", fs, flags, data); if (err) return err; It applies cleanly to both 3.13.x and 3.14.x and when applied the kernel prints the filesystems it tries to use against the specified block device. This way we get to see what actually happens. Here is how 3.13.11 does boot: [ 9.227459] md: Skipping autodetection of RAID arrays. (raid=autodetect will force) [ 9.233236] --> do_mount_root: name = /dev/root, fs = ext3, flags = 32768. [ 9.244404] --> do_mount_root: name = /dev/root, fs = ext2, flags = 32768. [ 9.248156] --> do_mount_root: name = /dev/root, fs = ext4, flags = 32768. [ 9.252284] --> do_mount_root: name = /dev/root, fs = fuseblk, flags = 32768. [ 9.256775] --> do_mount_root: name = /dev/root, fs = xfs, flags = 32768. [ 9.262779] --> do_mount_root: name = /dev/root, fs = btrfs, flags = 32768. [ 9.266784] btrfs: device fsid 80a18cd2-d804-4cc2-83e2-1286db96ecb4 devid 1 transid 61 /dev/root [ 9.274872] btrfs: disk space caching is enabled [ 9.460619] VFS: Mounted root (btrfs filesystem) on device 0:14. [ 9.465431] devtmpfs: mounted And here is what 3.14.1 does: [ 11.682145] md: Skipping autodetection of RAID arrays. (raid=autodetect will force) [ 11.685204] --> do_mount_root: name = /dev/root, fs = ext3, flags = 32768. [ 11.705999] --> do_mount_root: name = /dev/root, fs = ext2, flags = 32768. [ 11.709085] --> do_mount_root: name = /dev/root, fs = ext4, flags = 32768. [ 11.712267] --> do_mount_root: name = /dev/root, fs = fuseblk, flags = 32768. [ 11.715105] --> do_mount_root: name = /dev/root, fs = xfs, flags = 32768. [ 11.720494] VFS: Cannot open root device "sda2" or unknown-block(8,2): error -38 [ 11.723463] Please append a correct "root=" boot option; here are the available partitions: [ 11.727965] 0b00 1048575 sr0 driver: sr [ 11.730085] 0800 12582912 sda driver: sd [ 11.732023] 0801 51200 sda1 000095d3-01 [ 11.733742] 0802 5242880 sda2 000095d3-02 [ 11.734573] 0803 4194304 sda3 000095d3-03 [ 11.735406] 0804 3093504 sda4 000095d3-04 [ 11.736257] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block (8,2) [ 11.737585] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.14.1-mix64-debug-dirty #1 [ 11.738817] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2012 [ 11.740531] 0000000000008000 ffff88003f8e1dc8 ffffffff817b877a 0000000000000001 [ 11.742113] ffffffff819e9918 ffff88003f8e1e48 ffffffff817b40f3 ffffffff81c37dc0 [ 11.743498] ffff880000000010 ffff88003f8e1e58 ffff88003f8e1df8 0000000034616473 [ 11.744997] Call Trace: [ 11.745447] [<ffffffff817b877a>] dump_stack+0x46/0x58 [ 11.746318] [<ffffffff817b40f3>] panic+0xbc/0x1ba [ 11.747106] [<ffffffff81c9a3f7>] mount_block_root+0x206/0x2a4 [ 11.748053] [<ffffffff81c9a60b>] mount_root+0x57/0x5b [ 11.748920] [<ffffffff81c9a771>] prepare_namespace+0x162/0x19b The question is - why does not btrfs register in time? And for anyone wondering - btrfs is still there, built-in. Using the same kernel, only switching to XFS root (copied over from the btrfs one) - I get to mount the btrfs partition without problems, from userspace. Any ideas?
Created attachment 133741 [details] DEBUG: VISIBLY show tried filesystems during init->do_mount_root DEBUG: VISIBLY show tried filesystems during init->do_mount_root
Created attachment 133751 [details] dmesg output of patched 3.13.11 via serial console dmesg output of patched 3.13.11 via serial console
Created attachment 133761 [details] dmesg output of patched 3.14.1 via serial console dmesg output of patched 3.14.1 via serial console
Some discussion going on at linux-btrfs ML: GMANE link 1: http://news.gmane.org/find-root.php?group=gmane.comp.file-systems.btrfs&article=34445 GMANE link 2: http://thread.gmane.org/gmane.comp.file-systems.btrfs/34531/focus=34539
Marking this bug entry as RESOLVED because debugging led to the actual problem in do_mounts.c - some filesystems mount routines return error codes other than 0, EACCES and EINVAL and such return codes result in the kernel panicking without trying to mount root with all of the available filesystems. Patch is available as attachment to bug 74901.