Bug 61951

Summary: Negative Numbers shown in btrfs qgroup show output
Product: File System Reporter: dustymabe
Component: btrfsAssignee: Jan Schmidt (kernelbug)
Status: CLOSED PATCH_ALREADY_AVAILABLE    
Severity: normal CC: dsterba, kernelbug
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 3.10.10-200.fc19.x86_64 Tree: Mainline
Regression: No
Attachments: Partial fix for negative qgroup tracking numbers

Description dustymabe 2013-09-24 03:30:11 UTC
I am able to get negative numbers to get output from 'btrfs qgroup show' if I enable quota and  then delete a subvolume (in this case a snapshot). Note that this was done on Fedora 19 with kernel 3.10.10-200.fc19.x86_64 and btrfs-progs-0.20.rc1.20130308git704a08c-1.fc19.x86_64. I reported this issue to the mailing list and was asked to open a ticket. 

Steps are shown below:

[root@localhost ~]# mkfs.btrfs /dev/sdb

WARNING! - Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/sdb
        nodesize 4096 leafsize 4096 sectorsize 4096 size 10.00GB
Btrfs v0.20-rc1
[root@localhost ~]# 
[root@localhost ~]# mount /dev/sdb /btrfs/
[root@localhost ~]# mkdir /btrfs/.snapshots
[root@localhost ~]# 
[root@localhost ~]# cp /root/50M_File /btrfs/
[root@localhost ~]# btrfs subvolume snapshot /btrfs /btrfs/.snapshots/snap1
Create a snapshot of '/btrfs' in '/btrfs/.snapshots/snap1'
[root@localhost ~]#                                                                                                                                                                                            
[root@localhost ~]# cp /root/4151M_File /btrfs/                                                                                                                                                                
[root@localhost ~]# btrfs subvolume snapshot /btrfs /btrfs/.snapshots/snap2                                                                                                                                    
Create a snapshot of '/btrfs' in '/btrfs/.snapshots/snap2'                                                                                                                                                     
[root@localhost ~]#                                                                                                                                                                                            
[root@localhost ~]# btrfs quota enable /btrfs/                                                                                                                                                                 
[root@localhost ~]# btrfs qgroup show  /btrfs/                                                                                                                                                                 
0/5 4405821440 8192                                                                                                                                                                                            
0/256 52432896 4096                                                                                                                                                                                            
0/262 4405821440 8192                                                         
[root@localhost ~]#                                                           
[root@localhost ~]# rm /btrfs/4151M_File
[root@localhost ~]# btrfs qgroup show  /btrfs/                                
0/5 52432896 4096                                                             
0/256 52432896 4096                                                           
0/262 4405821440 4353392640
[root@localhost ~]# btrfs subvolume delete /btrfs/.snapshots/snap2 
Delete subvolume '/btrfs/.snapshots/snap2'
[root@localhost ~]# btrfs qgroup show  /btrfs/
0/5 -4300947456 -4353376256
0/256 52432896 4096
0/262 4405809152 4353380352
Comment 1 Josef Bacik 2013-09-24 13:06:03 UTC
I'm not able to reproduce this on either 3.10.7 or btrfs-next so it must be racy and specific to your setup.  How much ram do you have on this box, and what kind of disk is this?
Comment 2 dustymabe 2013-09-24 13:16:13 UTC
(In reply to Josef Bacik from comment #1)

This was a KVM guest with ~4G of ram and a virtio-scsi block device as sdb. I can try to get more details later when I get home if more details are needed. 

One thing to note is that it can take some time (a minute or so) before 'qgroup show' actually reports negative numbers. Initially the numbers stay the same.
Comment 3 Jan Schmidt 2013-10-17 06:57:30 UTC
I've got it reproduced, looking into it. (I cannot change the Status of this bug?)
Comment 4 David Sterba 2013-10-17 09:33:50 UTC
Reassigned.
Comment 5 Jan Schmidt 2013-10-22 09:15:53 UTC
Created attachment 111891 [details]
Partial fix for negative qgroup tracking numbers

Can you please try the attached patch and report if it helps? The fix is not qgroup-specific. I believe that it is correct as a generic fix and it passes the btrfs and generic xfstests for me.

Using this patch the large positive numbers for the deleted subvolume should be replaced by 0/0. The tracking numbers for the default subvolume go to -8KB where they really should be +4KB instead. The reason for that must be different, I'm looking into that next.
Comment 6 dustymabe 2013-10-22 12:48:35 UTC
(In reply to Jan Schmidt from comment #5)

> Can you please try the attached patch and report if it helps?

I'm not sure if you were asking me or someone else. If me, then I can try to get this up and running sometime later this week and report back. 

Dusty
Comment 7 Jan Schmidt 2013-10-22 12:52:17 UTC
Yes, I'm asking you. Thanks.
Comment 8 Jan Schmidt 2013-10-22 15:38:09 UTC
Bisected the second problem, it vanishes when one reverts this commit:

> commit 573aecafca1cf7a974231b759197a1aebcf39c2a
> Author: Josef Bacik <jbacik@fusionio.com>
> Date:   Fri Aug 30 14:38:49 2013 -0400
> 
>     Btrfs: actually limit the size of delalloc range

So probably something timing related on the qgroup tracking path. Looking further in that direction.
Comment 9 David Sterba 2013-10-22 22:54:14 UTC
(In reply to Jan Schmidt from comment #5)
> Created attachment 111891 [details]
> Partial fix for negative qgroup tracking numbers

I'll test the patch as well.
Comment 10 dustymabe 2013-10-23 15:48:23 UTC
I pulled the kernel sources for my current kernel from http://koji.fedoraproject.org/koji/buildinfo?buildID=460270 and rebuilt with the patch applied. After putting the new btrfs kernel module in place and running through the same steps I no longer see the problem. This is what I get at the end where I was previously getting negative numbers:

[root@localhost ~]# btrfs qgroup show  /btrfs/
0/5 52432896 4096
0/256 52432896 4096
0/263 0 0

Thus the "negative numbers" problem seems to be fixed. I'll let David possible test with a newer "upstream" kernel and verify what I have seen.

Thanks for the patch.
Comment 11 Jan Schmidt 2013-10-24 06:49:22 UTC
Thanks. I'm still looking into the -8KB I'm getting.
Comment 12 Jan Schmidt 2013-10-24 12:11:45 UTC
I've got that also, but it has nothing to do with the problem you were describing. As your bug is fixed, I'm closing this bug now. Patches to appear on the mailing list soon.

David: Please reopen if your test should come out negative.
Comment 13 David Sterba 2013-10-24 12:41:15 UTC
The reproducer from comment 0 does not lead to negative numbers with 3.12-rc6 here, not even after waiting until the deleted subvolume goes away completely. I'll check with an older kernel.
Comment 14 Jan Schmidt 2013-10-24 12:46:51 UTC
Reproducing instructions:

I don't think an older kernel will help. Whether this happens or not is timing-dependent. I use a slubdebug kernel to reproduce which makes things extremely slow. Makes sense as the OP uses a (presumably extremely slow) VM. The bug only happens when the extents are removed from the cleaner thread.
Comment 15 David Sterba 2013-10-24 13:34:56 UTC
Now reproduced with 3.12 (and with 3.11), I did not use a 4G file first.