Bug 101191 - balance failure following ext4 to btrfs conversion, BTRFS: error in btrfs_run_delayed_refs:2821: errno=-5 IO failure
Summary: balance failure following ext4 to btrfs conversion, BTRFS: error in btrfs_run...
Status: NEW
Alias: None
Product: File System
Classification: Unclassified
Component: btrfs (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Josef Bacik
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-07-08 06:23 UTC by Chris Murphy
Modified: 2017-04-28 08:33 UTC (History)
3 users (show)

See Also:
Kernel Version: 4.1.0-1.fc23.x86_64+debug
Tree: Fedora
Regression: No


Attachments
dmesg (54.51 KB, text/plain)
2015-07-08 06:23 UTC, Chris Murphy
Details
c1 dmesg (573.24 KB, text/plain)
2015-07-09 01:08 UTC, Chris Murphy
Details

Description Chris Murphy 2015-07-08 06:23:01 UTC
Created attachment 182141 [details]
dmesg

Similar setup to bug 101181 but the balance is done with kernel 4.1.

1. New ext4 system populate with a 1.5GB ISO and a Fedora 22 installation (mix of thousands of files). Conversion and balance done with btrfs-progs 4.1 and kernel 4.1 (debug).

[root@localhost ~]# blkid
/dev/vda1: UUID="fc4a811b-9612-4e87-841f-8a4f974df325" TYPE="ext4" PARTUUID="0ff5cebc-01"
/dev/vdb1: UUID="b2e2fcad-f66d-4fb0-a80d-7aa7d7939b25" TYPE="ext4" PARTUUID="0ff5cebc-01"
[root@localhost ~]# btrfs-convert /dev/vdb1
create btrfs filesystem:
	blocksize: 4096
	nodesize:  16384
	features:  extref, skinny-metadata (default)
creating btrfs metadata.
copy inodes [.] [    152846/    155714]
creating ext2fs image file.
cleaning up system chunk.
conversion complete.
[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# btrfs sub list /mnt
ID 256 gen 16 top level 5 path ext2_saved
[root@localhost ~]# btrfs sub del /mnt/ext2_saved/
Delete subvolume (no-commit): '/mnt/ext2_saved'
[root@localhost ~]# btrfs fi defrag -r /mnt
[root@localhost ~]# umount /mnt
[root@localhost ~]# btrfs check /dev/vdb1
Checking filesystem on /dev/vdb1
UUID: 9dfef243-60e1-499f-8e22-ba3e86df6f84
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 5914157064 bytes used err is 0
total csum bytes: 5117496
total tree bytes: 673841152
total fs tree bytes: 660176896
total extent tree bytes: 6504448
btree space waste bytes: 184429934
file data blocks allocated: 5245952000
 referenced 5245952000
btrfs-progs v4.1
[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# btrfs fi show /mnt
Label: none  uuid: 9dfef243-60e1-499f-8e22-ba3e86df6f84
	Total devices 1 FS bytes used 5.51GiB
	devid    1 size 40.00GiB used 26.03GiB path /dev/vdb1

btrfs-progs v4.1
[root@localhost ~]# btrfs fi df /mnt
Data, single: total=20.00GiB, used=3.96GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=6.00GiB, used=1.55GiB
GlobalReserve, single: total=128.00MiB, used=0.00B
[root@localhost ~]# btrfs balance start /mnt
ERROR: error during balancing '/mnt' - Read-only file system
There may be more info in syslog - try dmesg | tail
Comment 1 Chris Murphy 2015-07-09 01:07:06 UTC
Trying to reproduce this again, I'm getting a different result where btrfs balance hangs and I start getting multiple trace backs, with blocked tasks. Attaching the sequence from the shell here, and then attaching new dmesg2 that includes sysrq w and t:


[root@localhost ~]# blkid
/dev/vda1: UUID="fc4a811b-9612-4e87-841f-8a4f974df325" TYPE="ext4" PARTUUID="0ff5cebc-01"
/dev/vdb1: UUID="b2e2fcad-f66d-4fb0-a80d-7aa7d7939b25" TYPE="ext4" PARTUUID="0ff5cebc-01"
[root@localhost ~]# btrfs-convert /dev/vdb1
create btrfs filesystem:
	blocksize: 4096
	nodesize:  16384
	features:  extref, skinny-metadata (default)
creating btrfs metadata.
copy inodes [.] [    152740/    155714]
creating ext2fs image file.
cleaning up system chunk.
conversion complete.
[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# btrfs sub list /mnt
ID 256 gen 16 top level 5 path ext2_saved
[root@localhost ~]# btrfs sub del /mnt/ext2_saved/
Delete subvolume (no-commit): '/mnt/ext2_saved'
[root@localhost ~]# btrfs fi defrag -r /mnt
[root@localhost ~]# umount /mnt
[root@localhost ~]# btrfs check /dev/vdb1
Checking filesystem on /dev/vdb1
UUID: df4ca7c1-3088-499b-82c8-d8175bd70de1
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 5914157064 bytes used err is 0
total csum bytes: 5117496
total tree bytes: 673841152
total fs tree bytes: 660176896
total extent tree bytes: 6504448
btree space waste bytes: 184429909
file data blocks allocated: 5245952000
 referenced 5245952000
btrfs-progs v4.1
[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# btrfs balance start /mnt


...hang...  

so far this has been hung up for 10+ minutes and does not appear to complete. Control-C does not cancel and now there are no more kernel messages recurring.
Comment 2 Chris Murphy 2015-07-09 01:08:57 UTC
Created attachment 182221 [details]
c1 dmesg

This dmesg goes with comment 1. Actually there were too many messages to fit in the kernel buffer, so this was obtained using 'journalctl -k -l -o short-monotonic'
Comment 3 Chris Murphy 2015-07-09 01:21:36 UTC
[root@localhost ~]# btrfs check /dev/vdb1
Checking filesystem on /dev/vdb1
UUID: df4ca7c1-3088-499b-82c8-d8175bd70de1
checking extents
Aborted (core dumped)
[root@localhost ~]# btrfs-image -c9 -t4 /dev/vdb1 chris.btrfs
checksum verify failed on 37945344 found C87AFAD1 wanted 00002100
checksum verify failed on 37945344 found C87AFAD1 wanted 00002100
bytenr mismatch, want=37945344, have=553648170
Error reading metadata block
Error adding block -5
checksum verify failed on 37945344 found C87AFAD1 wanted 00002100
checksum verify failed on 37945344 found C87AFAD1 wanted 00002100
bytenr mismatch, want=37945344, have=553648170
Error reading metadata block
Error flushing pending -5
create failed (Success)

I have filed these btrfs-progs failures as bug 101221.
Comment 4 Chris Murphy 2015-07-09 02:44:14 UTC
Using -n|--no-inline when converting doesn't matter, subsequent balance still fails. http://www.spinics.net/lists/linux-btrfs/msg45037.html

Note You need to log in before you can comment on or make changes to this bug.